Introduction to MLSQL deep learning (4) -Serving

All code examples in this article are based on the latest version of MLSQL Engine 2.1.0-SNAPSHOT

This article uses a notebook demonstration of the MLSQL Console.

List of series articles:

1. MLSQL Machine Learning Minimalist Tutorial (No Python Required!)
2. MLSQL deep learning introduction [1]
3. Introduction to MLSQL deep learning [2] -Distributed model training
4. Introduction to MLSQL deep learning [3] -Feature engineering
5. Introduction to MLSQL deep learning [4] -Serving

For environmental requirements, please refer to the first article: Introduction to MLSQL deep learning [1].

Built-in model end-2-end http based service

Deploying the built-in model is very simple. When you have trained the algorithm model, including the feature engineer models, you can convert them all into UDF functions and deploy them to provide services online with one click.

The specific operation is as follows:

We deploy the model to the specified service by deployModel annotation and the register statement.

Now, we can use our predict service with http calling:

The above statement is equivalent to initiating a POST request.

See the MLSQL machine learning minimalist tutorial for details (no Python required!)

Deep learning end-2-end http based service

There are two steps to deploying deep learning as an HTTP service.

The first step is to deploy the model

The deployment method is also very simple, as follows:

First look at the notes:

--%deployModel
--%url=http://127.0.0.1:9003
--%access_token=mlsql

This is actually to deploy the specified model to the specified url service. How to deploy it? The user only needs to set three variables:

set model="ai_model.minist_model";
set localPath="/tmp/minist_model";
set  pythonEnv = "source /Users/allwefantasy/opt/anaconda3/bin/activate ray1.3.0";

The first parameter specifies the table name where the model is located. The second specifies the model to be placed in the local directory of the prediction service. After these values are set, the system will automatically deploy the model to the directory specified by the specified service.

The second step is to deploy the prediction script

The code is as follows:

Let’s look at the comments section firstly:

--%deployScript
--%url=http://127.0.0.1:9003
--%access_token=mlsql
--%codeName=testScript

We will deploy the script to the specified Url service. And we have named the script, called testScript. Now look at the script code section:

First, get the parameters of the http interface in the following way. This is fixed.

params = dict([(row["key"],row["value"]) for row in context.fetch_once_as_rows()])

The location of the model is fixed because we have determined it when we publish the model. So you can load it directly:

localPath="/tmp/minist_model"
model = models.load_model(localPath)

If the result is output, use the following method:

context.build_result([{"value":value}])

The output must be in the format above, where value is a json string.

After successful registration, the system returns the predicted address.

Test the interface

We can test the interface, here we use crawler_http function to access the interface:

The system successfully loaded the model, so the result can be returned smoothly. The verification was successful.

Is the model serve open source?

Yes. MLSQL Engine is also a Model Serving Engine. When MLSQL Engine is deployed on a stand-alone basis, it is consistent with standard Java programs. We have developed built-in plug-ins to support the prediction and execution of python code. For example, the above example code has an execution time of less than 10ms.

Finally, how to avoid loading the model every time

The above code actually loads the model every time it is executed, which is obviously not in line with the actual production. We can make fewer changes to ensure that it is loaded only once.

#%deployScript
#%url=http://127.0.0.1:9003
#%access_token=mlsql
#%codeName=testScript
import json
from pyjava.api.mlsql import RayContext,PythonContext
from tensorflow.keras import models,layers
from tensorflow.keras import utils as np_utils
params = dict([(row["key"],row["value"]) for row in context.fetch_once_as_rows()])
localPath="/tmp/minist_model"
codeName = params["codeName"]
if "models" not in PythonContext.cache:
PythonContext.cache["models"]={}
if codeName not in PythonContext.cache["models"]:
PythonContext.cache["models"][codeName]=models.load_model(localPath)
model= PythonContext.cache["models"][codeName]
data=params["data"]
value=json.dumps({"hello":data})
context.build_result([{"value":value}])