Serialize figures without deep copying them, add a msgspec engine

CHANGELOG.md

@@ -4,6 +4,12 @@ This project adheres to [Semantic Versioning](http://semver.org/).
 
 ## Unreleased
 
+### Added
+- Add a `msgspec` JSON engine, used by the `auto` engine when `msgspec` is installed and `orjson` is not [[#5813](https://github.com/plotly/plotly.py/pull/5813)]
+
+### Updated
+- Speed up `plotly.io.to_json` and `PlotlyJSONEncoder` for figures and for payloads that mix plotly objects with plain data: figures are no longer deep copied, and with orjson only the values orjson can't serialize go through the slow cleaning path [[#5813](https://github.com/plotly/plotly.py/pull/5813)]
+
 ### Fixed
 - Fix `mpl_to_plotly` tick marker mirroring, axis positioning, and trace visibility for twinned and overlaid subplots (such as `twinx` and `twiny`) [[#5310](https://github.com/plotly/plotly.py/pull/5310)], with thanks to @robertoffmoura for the contribution!
 - Fix `mpl_to_plotly` losing matplotlib axis line colors by exporting the color of each axis's displayed spine to the plotly x and y axis `linecolor` [[#5311](https://github.com/plotly/plotly.py/pull/5311)], with thanks to @robertoffmoura for the contribution!

_plotly_utils/utils.py

@@ -1,4 +1,5 @@
 import base64
+import datetime
 import decimal
 import json as _json
 import sys
@@ -114,6 +115,40 @@ def convert_to_base64(obj):
             convert_to_base64(value)
 
 
+def to_base64_copy(obj):
+    """
+    Like convert_to_base64, but leaves obj untouched: only the containers
+    that change are copied, everything else is shared with obj.
+    """
+    if isinstance(obj, dict):
+        out = None
+        for key, value in obj.items():
+            if is_skipped_key(key):
+                continue
+            if is_homogeneous_array(value):
+                new = to_typed_array_spec(value)
+            else:
+                new = to_base64_copy(value)
+            if new is not value:
+                if out is None:
+                    out = dict(obj)
+                out[key] = new
+        return obj if out is None else out
+    elif isinstance(obj, (list, tuple)):
+        # Data arrays hold scalars: skip them without a Python-level loop.
+        if not any(issubclass(t, (dict, list, tuple)) for t in set(map(type, obj))):
+            return obj
+        out = None
+        for i, value in enumerate(obj):
+            new = to_base64_copy(value)
+            if new is not value:
+                if out is None:
+                    out = list(obj)
+                out[i] = new
+        return obj if out is None else out
+    return obj
+
+
 def cumsum(x):
     """
     Custom cumsum to avoid a numpy import.
@@ -218,6 +253,16 @@ def default(self, obj):
         Therefore, we only anticipate either unknown iterables or values here.
 
         """
+        # Only the base class takes these shortcuts: subclasses may override
+        # the encode_as_* methods.
+        # Exact types only: pandas.Timestamp and pandas.NaT subclass datetime.
+        if type(self) is PlotlyJSONEncoder:
+            if type(obj) in (datetime.datetime, datetime.date):
+                return obj.isoformat()
+            to_json_dict = getattr(obj, "_to_json_dict", None)
+            if to_json_dict is not None:
+                return to_json_dict()
+
         # TODO: The ordering if these methods is *very* important. Is this OK?
         encoding_methods = (
             self.encode_as_plotly,

doc/python/renderers.md

@@ -271,6 +271,6 @@ No matter the approach chosen to display a figure, [the figure data structure](h
 
 *New in v5.0*
 
-The default JSON serialization mechanism can be slow for figures with many data points or with large `numpy` arrays or data frames. **If [the `orjson` package](https://github.com/ijl/orjson) is installed**, `plotly` will use that instead of the built-in `json` package, which can lead to **5-10x** speedups for large figures.
+The default JSON serialization mechanism can be slow for figures with many data points or with large `numpy` arrays or data frames. **If [the `orjson` package](https://github.com/ijl/orjson) is installed**, `plotly` will use that instead of the built-in `json` package, which can lead to **5-10x** speedups for large figures. If `orjson` isn't installed but [`msgspec`](https://github.com/jcrist/msgspec) is (Dash installs it), `plotly` uses `msgspec`, which gives most of the same speedup.
 
 Once a figure is serialized to JSON, it must be rendered by a browser, either immediately in the user's browser, at some later point if the figure is exported to HTML, or immediately in Kaleido's internal headless browser for static image export. Rendering time is generally proportional to the total number of data points in the figure, the number of traces and the number of subplots. In situations where rendering performance is slow, we recommend considering [the use of `plotly` WebGL traces](/python/webgl-vs-svg/) to exploit GPU-accelerated rendering in the browser, or [using the Datashader library to do Python-side rendering](/python/datashader/) before using `px.imshow()` to render the figure.

plotly/basedatatypes.py

@@ -16,6 +16,7 @@
     chomp_empty_strings,
     find_closest_string,
     convert_to_base64,
+    to_base64_copy,
 )
 from _plotly_utils.exceptions import PlotlyKeyError
 from .optional_imports import get_module
@@ -3323,6 +3324,23 @@ def to_dict(self):
 
         return res
 
+    def _to_json_dict(self):
+        """
+        to_dict() without the deepcopy, for serializers that only read it.
+        Shares data with the figure: never mutate the result.
+        """
+        cls = type(self)
+        if (
+            cls.to_dict is not BaseFigure.to_dict
+            or cls.to_plotly_json is not BaseFigure.to_plotly_json
+        ):
+            return self.to_plotly_json()
+        res = {"data": self._data, "layout": self._layout}
+        frames = [frame._props for frame in self._frame_objs]
+        if frames:
+            res["frames"] = frames
+        return to_base64_copy(res)
+
     def to_plotly_json(self):
         """
         Convert figure to a JSON representation as a Python dict
@@ -3440,6 +3458,7 @@ def to_json(self, *args, **kwargs):
             The JSON encoding engine to use. One of:
               - "json" for an encoder based on the built-in Python json module
               - "orjson" for a fast encoder the requires the orjson package
+              - "msgspec" for a fast encoder that requires the msgspec package
             If not specified, the default encoder is set to the current value of
             plotly.io.json.config.default_encoder.
 
@@ -3500,6 +3519,7 @@ def write_json(self, *args, **kwargs):
             The JSON encoding engine to use. One of:
               - "json" for an encoder based on the built-in Python json module
               - "orjson" for a fast encoder the requires the orjson package
+              - "msgspec" for a fast encoder that requires the msgspec package
             If not specified, the default encoder is set to the current value of
             plotly.io.json.config.default_encoder.
 
@@ -5562,6 +5582,12 @@ def on_change(self, callback, *args, **kwargs):
         # -----------------
         self._change_callbacks[arg_tuples].append(callback)
 
+    def _to_json_dict(self):
+        # to_plotly_json() without the deepcopy; never mutate the result
+        if type(self).to_plotly_json is not BasePlotlyType.to_plotly_json:
+            return self.to_plotly_json()
+        return self._props if self._props is not None else {}
+
     def to_plotly_json(self):
         """
         Return plotly JSON representation of object as a Python dict
@@ -5596,6 +5622,7 @@ def to_json(self, *args, **kwargs):
             The JSON encoding engine to use. One of:
               - "json" for an encoder based on the built-in Python json module
               - "orjson" for a fast encoder the requires the orjson package
+              - "msgspec" for a fast encoder that requires the msgspec package
             If not specified, the default encoder is set to the current value of
             plotly.io.json.config.default_encoder.