driver/CHANGELOG.md

Changelog

Unreleased

0.2.9 - 2024-08-16

Added

• Debug logging on columns read/write. Solves issue #404.
• Separate parse_url function for external tools. Solves issue #438.
• [NumPy] Support for pandas StringArray and ArrowStringArray writing. Solves issue #337. Pull request #441 by Simon-Chenzw.

Fixed

• ssl.PROTOCOL_TLS deprecation warning for Python 3.10+.
• Trailing spaces in external tables declaration. Solves issue #385.
• [Tests] Skip certificate validation test during run on local machine. Solves issue #442.
• Handle special enum values: 'mro' and ''. Solves issue #333.
• [NumPy] Do not localize already localized pandas datetime. Solves issue #444.
• [Tests] Run tests for server 24+.

Changed

• Protocol version bumped to 54468.

0.2.8 - 2024-06-12

Added

• Lazy date LUT initialization for memory reducing. See docs. Solves issue #430. Pull request #431 by DaniilAnichin.
• Dependabot for up-to-date GitHub actions. Pull request #421 by cclauss.

Fixed

• TCP keepalive idle_time_sec, interval_sec type to integer. Solves issue #432.
• Empty Map type read IndexError. Solves issue #433.
• Test on current versions of PyPy. Pull request #424 by cclauss.
• Connection verify=False ignoring. Solves issue #420. Pull request #421 by akurdyukov.

0.2.7 - 2024-02-20

Added

• Wheels for Python 3.12.
• Toggle hostname verification with verify option. Solves issue #378. Pull request #379 by adamleko.

Fixed

• Date32 start interval changed to 1900-01-01. Solves issue #409.
• Memory leak when datastore raise exception on BufferedSocketWriter.write_into_stream. Solves issue #406. Pull request #407 by pulina.
• input_format_null_as_default option for UUID produce 00000000-0000-0000-0000-000000000000 if set to true. Solves issue #401.
• [Tests] Remove MemoryTracker asserting on INSERT statements. Solves issue #403.
• Store "progress" and "profile" stats on INSERT statements. Solves issue #391. Pull request #392 by insomnes.
• Add send_logs_level=test log level support. Solves issue #383. Pull request #395 by the-horhe.
• self in int128_from_quads, int128_to_quads, int256_from_quads, int256_to_quads. Solves issue #400.

Changed

• Server-side parameters substitution is turned off by default. You can't mix client-side and server-side formatting in one query. Solves issue #376 and #410.
• Protocol version bumped to 54462.

0.2.6 - 2023-05-02

Added

• JSON type. Solves issue #320.
• On demand client revision downgrading.
• Server-side query templating.
• Sparse data type deserialization.
• TCP keepalive.
• [NumPy] Optional dataframe column names replacing. Pull request #361 by notsovitalik.
• Substitution for parameters with time type. Solves issue #359. Pull request #360 by ghazi-git.

Fixed

• Client construction with round_robin. Solves issue #373.
• Column state prefixes reading and writing. Solves issue #372.
• Inserts to a nullable LowCardinality columns. Solves issue #363. Pull request #366 by Dmitry-k42.
• [NumPy] Avoid unnecessary timezone conversion. Solves issue #354. Pull request #355 by edwinwzhe.

Changed

• Protocol version bumped to 54459.
• [NumPy] Speed-up reading Datetime64. Pull request #365 by joelgibson.

Removed

• Python 3.6 support.

0.2.5 - 2022-11-27

Added

• [NumPy] More readable exception for less columns in frame. Solves issue #320.
• Support server_hostname via SNI when connecting. Pull request #325 by dspangen.
• [NumPy] Bool support. Pull request #332 by DylanModesitt.
• Multiple connections round-robin.
• Wheels for Python 3.11.
• Docstring for substitute_params. Pull request #347 by jasonho-lynx.

Fixed

• Decimal support inside Map type. Pull request #297 by zaius.
• Preserve empty dataframe shape. Solves issue #321.
• Preserve nanoseconds on INSERT for DateTime64. Solves issue #307.
• Handle "progress" packets while inserting. Solves issue #326. Pull request #327 by carlosfy.
• [NumPy] Don't raise exception on INSERT when dataframe columns are superset of table's. Pull request #340 by andrewresnikoff.
• Re-cythonize files for Python 3.11. Pull request #342 by asottile-sentry.
• NULL defaults to empty Array instead of one-element array. Solves issue #339.
• Handle "profile event" packets while inserting.

Removed

• Python 3.5 support.

0.2.4 - 2022-06-13

Added

• dict and namedtuple cursor factories. Solves issue #290.
• Geo types: Point, Ring, Polygon, MultiPolygon. Solves issue #228.
• [NumPy] Tuple type.
• input_format_null_as_default option for NULL insert into not NULL columns. Solves issue #312.
• Chunk size controlling in execute_iter by parameter chunk_size. Solves issue #314. Pull request #315 by MIracleyin.

Fixed

• Decimal precision check on INSERT. Solves issue #289. Pull request #295 by joelynch.
• Using server's default database instead or default. Solves issue #309.
• python_requires to valid PEP 440 syntax in setup.py. Pull request #291 by LSturtew.

Changed

• Protocol version bumped to 54453.

0.2.3 - 2022-02-07

Added

• tzlocal>=4.0 support. Pull request #263 by azat.
• quota_key support.
• Wheels for Python 3.10.
• Bool type. Pull request #279 by adrian17.
• Nested type with flatten_nested=0. Pull request #285 by spff.

Fixed

• Handle partially consumed query. Solves issue #117.
• Fallback to generic columns when NumPy support is not implemented for column type. Solves issue #254.
• Broken ZSTD decompression. Solves issue #269.
• External tables passing with NumPy. Solves issue #267.
• Consider tzinfo for datetime parameters substitution. Solves issue #268.
• Do not use NumPy columns inside generic columns. Solves issue #272.
• Decimal128 and Decimal256 types_check. Solves issue #274.
• Insertion using execute in DB API. Solves issue #179. Pull request #276 by nnseva.
• Variables cannot be declared with cpdef in Cython 3. Pull request #281 by ym.

Changed

• Switch from nose test runner to pytest.
• Migrate from Travis CI to GitHub Actions.

0.2.2 - 2021-09-24

Added

• DateTime64 extended range. Pull request #222 by 0x4ec7.
• Support for using Client as context manager closing connection on exit. Solves issue #237. Pull request #206 by wlhjason.
• OpenTelemetry support. Solves issue #230.
• tzlocal>=3.0 support.
• Date32 type.
• [NumPy] Nullable(T) support.

Fixed

• Boxing on DataFrames INSERT.
• Empty external tables sending. Solves issue #240.
• Create error message before disconnect. Pull request #247 by NikiforovG.
• Writing explicit NumPy NaN values. Solves issue #249.
• UInt128 INSERTing. Solves issue #251.
• Large DataFrames chunking. Solves issue #243.
• Allow arbitrary DataFrame columns order on INSERT. Solves issue #245.

Changed

• Protocol version bumped to 54442.
• Unpin lz4 for CPython. Pin lz4<=3.0.1 only for PyPy.

Removed

• transpose parameter from insert_dataframe.

0.2.1 - 2021-06-02

Added

• Linux wheels for AArch64. Pull request #197 by odidev.
• [U]Int28/256 types.
• Decimal256 type.
• Code coverage for cython code.
• Map type.
• Support for private key/certificate file. Pull request #219 by alexole.

Fixed

• Empty Array(Tuple(T)) writing. Solves issue #177.
• Preserve Decimal precision on INSERT. Solves issue #192.
• Remove closed cursors from connection. Solves issue #194.
• DB API connect with default params.
• Fix log blocks handling. Solves issue #203.
• Multidimensional Array decoding. Solves issue #204. Pull request #206 by smagellan.
• Use last database after reconnect. Solves issue #205.
• Decimal(N, 1) handling. Pull request #210 by raw0w.

Changed

• Decimal128 performance increased (up to 25% compared to 0.2.0 release).

Removed

• Python 2.7 support.

0.2.0 - 2020-12-14

Added

• NumPy reading/writing for columns: Float32/64, [U]Int8/16/32/64, Date/DateTime(‘timezone’)/DateTime64(‘timezone’), String/FixedString(N), LowCardinality(T). Merge Arturus's fork.
• Pandas DataFrame reading/writing.
• Ability to mark all settings as important to fail on unknown settings on sever side.
• SSL SNI support. Solves issue #172.
• Wheels for Python 3.9 and PyPy.
• Remember last successful host on connection. Solves issue #168.

Fixed

• Server logs displaying on INSERT.
• Make exceptions picklable. Pull request #169 by azat.
• Enum type deserializing when it wrapped in SimpleAggregateFunction. Pull request #170 by flyAwayGG.
• Pin major tzlocal version. Solves issue #166.

Changed

• String and DateTime columns writing optimization.
• Array columns reading/writing optimization.
• Chunking optimization for large lists/tuples.
• Protocol version bumped to 54441.

0.1.5 - 2020-09-19

Added

• Do not require settings declaration if server support setting-as-string. Pull request #142 by azat.
• host_name in logs. Pull request #144 by azat.
• Cursor attribute columns_with_types to DB API. Issue #149.
• Cursor method set_query_id to DB API. Issue #152.

Fixed

• Connection error messages formatting.
• Client.from_url credentials unquoting. Issue #146.
• Empty nested array handling. Pull request #161 by dourvaris.
• read_varint overflow. Issue #163.
• Malformed reads/writes in BufferedReader. This addresses CVE-2020-26759.

Changed

• Use deque for ~4x speedup when reading Array columns. Pull request #164 by dourvaris.

0.1.4 - 2020-06-13

Added

• Tuple type.
• Custom String column encoding.
• Settings update to v20.4.1.2742 server version. Pull request #133 by azat.
• Settings update to v20.5.1.3657 server version. Pull request #141 by azat.
• Unsupported server versions to documentation.
• Performance section to documentation.
• Python 3.9 in Travis CI build matrix.

Fixed

• Reading/writing Array(Tuple).
• 20.x server version support.
• Settings mutation in execute.
• Slow columnar results returning (columnar=True).
• Segfault on passing not encoded strings during INSERT into ByteString column.

Changed

• Miscellaneous read/write optimizations
• Protocol version bumped to 54429.

0.1.3 - 2020-02-21

Added

• Python DB API 2.0.
• Multiple hosts support on connection errors.
• Insert columnar data support. Pull request #122 by Anexen.
• Wheels for Python 3.8.
• DateTime64 type.
• Settings update to v20.2.1.2201 server version. Pull request #123 by azat.

Fixed

• Client.from_url settings detection.
• Close socket on KeyboardInterrupt while running query.
• Null handling in LowCardinality columns.

Changed

• Protocol version bumped to 54421.
• Increased speed (up to 20-30% compared to 0.1.2 release) on heavy SELECT and INSERT queries. Pull request #122 by Anexen.
• Memory consumption decreased (up to 20% compared to 0.1.2 release). Pull request #122 by Anexen.

0.1.2 - 2019-10-18

Added

• Settings update to 19.16.1 server version. Pull request #111 by azat.
• Python 3.8 in Travis CI build matrix.
• Returning inserted rows count on INSERT queries with data. Returning rows count from INSERT FROM SELECT is not supported.

Fixed

• Exposing columnar parameter to execute_with_progress. Pull request #108 by igorbb.
• LowCardinality tests. Pull request #112 by azat.

Changed

• Increased speed (up to 5 times compared to 0.1.1 release) of INSERT queries.
• Date/DateTime columns selecting and inserting optimizations.

0.1.1 - 2019-09-20

Added

• Client.from_url method that creates client configured from the given URL.

Fixed

• If source column was timezone-aware values from DateTime column are returned with timezone now.
• Handling zero bytes in the middle of FixedString column. Issue #104.

0.1.0 - 2019-08-07

Added

• SimpleAggregateFunction type. Pull request #95 by azat.

Changed

• Increased speed (5-6 times compared to 0.0.20 release) of SELECT queries with large amount of strings.
• Package is distributed in source and binary forms now. Compilation from source is required for platforms without wheels.

Fixed

• Elapsed time calculation on INSERT.
• Dependencies environment markers for poetry in setup.py. Pull request #96 by nitoqq.

0.0.20 - 2019-06-02

Added

• LowCardinality(T) type.
• Access for processed rows, bytes and elapsed time of the last executed query.
• Allow to insert datetime into Date column. Pull request #75 by gle4er.
• 'max_partitions_per_insert_block' setting. Pull request #85 by mhsekhavat.

Fixed

• Fallback for user name if it's not defined. Pull request #87 by wawaka.

0.0.19 - 2019-03-31

Added

• IPv4/IPv6 types. Pull request #73 by AchilleAsh.

Fixed

• String enums escaping.

0.0.18 - 2019-02-19

Fixed

• Strings mishandling read from buffer. Pull request #72 by mitsuhiko.

0.0.17 - 2019-01-09

Added

• Server logs displaying.
• Documentation on Read the Docs: https://datastore-driver.readthedocs.io

Changed

• Protocol version bumped to 54406.

Fixed

• INSERT generators support as data parameter.
• INSERT null value on ByteString column. Pull request #65 by vivienm.
• Integer types support in FloatColumn.
• Handle quotes and equation signs in Enum options. Pull request #67 by sochi.

0.0.16 - 2018-10-09

Added

• Option to access profile info about the last executed query. Pull request #57 by vanzi.
• Decimal type.
• Settings update to 18.14.2 server version.

Changed

• String column read/write optimizations.
• Protocol version bumped to 54401.

Fixed

• Client settings are not mutable now.

0.0.15 - 2018-09-26

Fixed

• Unpin datastore-cityhash dependency.

0.0.14 - 2018-08-16

Added

• Block by block results streaming.

0.0.13 - 2018-07-26

Fixed

• Revert pull request #45 by shotInLeg due to INSERT FROM SELECT breaking.

0.0.12 - 2018-07-25

Added

• Allow for access of final progress stats. Pull request #41 by alex-hofsteede.
• Supplying raw integers in DateTime columns. Pull request #42 by wawaka.

Fixed

• Pip install in editable mode.

Changed

• Raise ValueError instead of timeout for INSERT queries without params. Pull request #45 by shotInLeg.

0.0.11 - 2018-06-03

Added

• Timezone support in DateTime type.
• Python 3.7 and PyPy in Travis CI build matrix.
• Direct bytes support in FixedString. Pull request #26 by lidalei.

Removed

• Python 3.3 support.

0.0.10 - 2018-03-14

Added

• Server version specific tests.
• Nothing type.
• Travis CI build matrix.

Fixed

• Possible IndexError in packet types string representation. Pull request #28 by WouldYouKindly.
• Do not use timezone in Date columns. Issue #29.
• Approximate rows to read calculation in execute_with_progress. Pull request #30 by b1naryth1ef.

Changed

• List and Tuple types rendering in parameters substitution. Pull request #27 by silentsokolov.

0.0.9 - 2018-01-17

Added

• Interval type. Pull request #16 by kszucs.

Fixed

• Raise EOFError when no data is read in _read_one in Python 3.
• SSL file descriptors closing in Python 2.7.

Changed

• Log level for normal operations raised to DEBUG. Pull request #17 by kmatt.

Removed

• Drop Python 3.2 support due to lz4 issues.

0.0.8 - 2017-10-23

Added

• Parameters substitution for SELECT queries.
• SSL support.

Fixed

• Columnar result returning from multiple blocks. Columns must be concatenated.
• Settings logger was root.
• Reading/writing Array(String) NotImplementedError raise.
• IPv6 support issue #12.

0.0.7 - 2017-10-12

Added

• Configurable logging level in tests.
• Full error codes list.
• Force check datastore-cityhash is installed if compression is used.
• Client can be directly imported from package.
• insert_block_size parameter - maximum rows in block (default is 1048576).
• Columnar result returning (columnar=True). Pull request #11 by kszucs.
• Tunable types check (types_check=True). Off by default.

Changed

• Handling only socket-related errors on ping. Errors are logged with WARNING level.
• Client is created per one test.
• Sending/receiving data speed significantly increased.

Fixed

• Version detection in setup.py.
• Error handling on socket.shutdown.
• Install enum34 only if required.
• datastore-cityhash import issue #10.

Removed

• QuickLZ support
• six dependency

0.0.6 - 2017-09-19

Added

• UUID type.
• Query limits settings.
• Code coverage.
• Datastore server and driver version upped to 54276.
• Changelog.
• Added column name to TypeMismatchError.
• Query progress information.
• Version of package.

Fixed

• socket.timeout error handling on connect.

0.0.5 - 2017-07-16

Added

• Nullable(T) type.
• Return data from TOTALS and EXTREMES packets.
• Query settings.
• query_id execution option.
• NULL type.
• Raise exception on SELECT queries.

Changed

• Small columns refactoring.
• datastore-client in tests moved to docker.

0.0.4 - 2017-06-15

Added

• FixedString(N) type.
• Enum8/16 types.
• Array(T) type.
• External data for query processing
• Raise UnknownTypeError for unsupported columns.

Changed

• Socket connect timeout fix.

0.0.3 - 2017-05-24

Added

• QuickLZ, LZ4/LZ4HC, ZSTD compressions.
• Support old servers without BlockInfo.
• Travis CI.
• flake8 syntax check.

0.0.2 - 2017-05-16

Added

• [U]Int8/16/32/64 types.
• Date/DateTime types.
• String types.