Machine Learning with scikit-learn
Andreas Mueller
Overview
- Lightning tour of scikit-learn
- Latest goodies in stable (0.14)
- GridSearchCV with arbitrary metrics
- Randomized Search
- Partial Fit in Naive Bayes
- (Feature Imputation)
- (AdaBoost)
- (RBMs)
- ...
- Upcoming goodies in next release (0.15)
- Way better trees and forests
- Sparse support for trees (?)
- make_pipeline, make_union
- (Ransac)
In [1]:
from sklearn.datasets import load_digits
digits = load_digits()
X, y = digits.data, digits.target
print(X.shape)
print(y.shape)
plt.matshow(X[0, :].reshape(8, 8))
Out[1]:
In [2]:
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y)
from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(n_estimators=50)
rf.fit(X_train, y_train)
print(rf.score(X_test, y_test))
In [3]:
from sklearn.grid_search import GridSearchCV
search = GridSearchCV(rf, param_grid={'max_depth': [1, 3, 5, 10]}, cv=5)
search.fit(X_train, y_train)
print(search.score(X_test, y_test))
0.14 Cherries
Grid Search with Arbitrary Metrics
In [4]:
import pandas as pd
df_train = pd.read_csv("../../biology_datasets_stuff/AID687red_train.csv")
print(df_train.Outcome.value_counts())
X_train, y_train = df_train.values[:, :-1], df_train.values[:, -1]
y_train = y_train == 'Active'
print(X_train.shape)
print(y_train.shape)
In [5]:
search = GridSearchCV(rf, param_grid={'max_depth': [1, 3, 5, 10]}, cv=3, verbose=10)
search.fit(X_train, y_train)
Out[5]:
In [6]:
search = GridSearchCV(rf, param_grid={'max_depth': [1, 3, 5, 10]}, scoring='roc_auc', cv=3, verbose=10)
search.fit(X_train, y_train)
Out[6]:
In [7]:
param_grid = {'max_depth': [1, 3, 5, 10]}
GridSearchCV(rf, param_grid, scoring='accuracy')
GridSearchCV(rf, param_grid, scoring='roc_auc')
GridSearchCV(rf, param_grid, scoring='average_precision');
In [8]:
from sklearn.metrics import SCORERS
for scoring in SCORERS:
print(scoring)
Defining your own scorer
In [9]:
from sklearn.metrics import make_scorer
def my_freaky_loss(y_true, probability):
return np.sum(probability[np.arange(len(y_true)), y_true] > .4)
my_freaky_scorer = make_scorer(my_freaky_loss, needs_probability=True)
GridSearchCV(rf, param_grid, scoring=my_freaky_scorer)
def freaky_regularized_scorer(estimator, X, y_true):
return estimator.score(X, y_true) + .1 * np.linalg.norm(estimator.coef_)
GridSearchCV(rf, param_grid, scoring=freaky_regularized_scorer);
Randomize Parameter Search
In [10]:
from sklearn.pipeline import Pipeline
from sklearn.feature_selection import SelectKBest
from sklearn.svm import SVC
from sklearn.grid_search import RandomizedSearchCV
from scipy.stats import expon
X, y = digits.data, digits.target
pipe = Pipeline([("feature_selection", SelectKBest()), ("classifier", SVC())])
param_distribution = {'feature_selection__k': np.arange(1, X.shape[1]), 'classifier__C': expon(0, 1),
'classifier__gamma': expon(0, 1), 'classifier__kernel': ['linear', 'rbf', 'poly'],
'classifier__degree': [1, 2, 3, 4]}
search = RandomizedSearchCV(pipe, param_distribution, verbose=10)
search.fit(X, y)
print(search.best_params_)
Partial Fit in Naive Bayes
In [11]:
from sklearn.naive_bayes import MultinomialNB
nb = MultinomialNB()
for i in range(10):
y_train_partial = np.load("y_train_%d.npy" % i)
X_train_partial = np.load("X_train_%d.npy" % i)
nb.partial_fit(X_train_partial, y_train_partial, classes=np.arange(10))
X_test, y_test = np.load("X_test.npy"), np.load("y_test.npy")
nb.score(X_test, y_test)
Out[11]:
0.15 crowd pleasers
Threading in Joblib
Aka: Trees in Random Forsts share
In [12]:
results = Parallel(n_jobs=self.n_jobs, verbose=self.verbose, backend="threading")(
delayed(_parallel_apply)(tree, X) for tree in self.estimators_)
More good stuff in tree models
- Support for C and Fortran arrays in trees.
- Faster mse computation.
- max_leaf_node stopping criterion for trees.
- feature caching for Extra Trees.
- some improvements to gradient boosting.
- 4-8x speed improvement for parallel computations!
Sparse Matrix Support for Tree Models
(maybe)
Sugar
In [13]:
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.svm import LinearSVC
# before:
from sklearn.pipeline import Pipeline, FeatureUnion
pipe = Pipeline([("features", FeatureUnion([("word_ngrams", CountVectorizer()), ("char_ngrams", CountVectorizer(analyzer="char"))])),
("svm", LinearSVC())])
print(pipe)
# now:
from sklearn.pipeline import make_pipeline, make_union
pipe = make_pipeline(make_union(CountVectorizer(), CountVectorizer(analyzer="char")),
LinearSVC())
print(pipe)
Thanks to
| @ogrisel | @GaelVaroquaux |
| @larsmans | @glouppe |
| @pprett | @vene |
| @mblondel | @agramfort |
| @arjoly | @jaquesgrobler |
| @jakevdp | @robertlayton |
| @NelleV | @ndawe |
| @jnothmann |
and many more....
| t3kcit@gmail.com |
| @t3kcit |
| @amueller |
| peekaboo-vision.blogspot.com |