Exception while reading from stream postgresql как исправить

Exception message: Exception while reading from stream

@arunsaho there are no details above — an exception message isn’t enough for us to help you. Please post a runnable code sample which triggers the issue, or at least provide more context.

string connString =
String.Format(
«Server={0};Username={1};Database={2};Port={3};Password={4};SSLMode=Prefer;Timeout=300;CommandTimeout = 300»,
Host,
User,
DBname,
Port,
Password);

        using (var conn = new NpgsqlConnection(connString))
        {
            Console.Out.WriteLine("Opening connection");
            
            try
            {
                conn.Open();
                
            using (var command = new NpgsqlCommand("INSERT INTO table1 (name1, dept, key) VALUES (@n1, @p1, @q1)", conn))
            {
                command.Parameters.AddWithValue("n1", name1);
                command.Parameters.AddWithValue("p1", dept);
                command.Parameters.AddWithValue("q1", key);

                int nRows = command.ExecuteNonQuery();
            }
            conn.Close();
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex);
                Console.WriteLine("Unable to Open Connection");
            }
        }

Exact stackTrace:

Npgsql.NpgsqlException (0x80004005): Exception while reading from stream
—> System.TimeoutException: Timeout during reading attempt
Npgsql.Internal.NpgsqlReadBuffer.g__EnsureLong|41_0(NpgsqlReadBuffer buffer, Int32 count, Boolean async, Boolean readingNotifications)
Npgsql.Internal.NpgsqlConnector.RawOpen(SslMode sslMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken, Boolean isFirstAttempt)
Npgsql.Internal.NpgsqlConnector.g__OpenCore|191_1(NpgsqlConnector conn, SslMode sslMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken, Boolean isFirstAttempt)
Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
Npgsql.ConnectorPool.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
Npgsql.ConnectorPool.g__RentAsync|28_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
Npgsql.NpgsqlConnection.g__OpenAsync|45_0(Boolean async, CancellationToken cancellationToken)
Npgsql.NpgsqlConnection.Open()

@arunsaho a network timeout during Open can occur for various reasons: you may have a firewall rule preventing the connection, there may be a misconfiguration, or if this happens intermittently, your network could be a bit flaky. In any case, this isn’t really related to Npgsql in any way — Npgsql simply establishes a standard TCP connection to the host and port you provide it.

If the error is reproducible and not intermittent, I’d recommend trying to connect to the same PostgreSQL (same host/port) from the same client machine where Npgsql is running, but from an external tool, e.g. the psql command-line tool. That would help isolate things.

Hello All — I’m getting a very similar error. We are using .net 6.0.7 Npgsql 6.0.5 AWS Arora Postgres 12.7 with RDS proxy. It is appearing in both our staging and prod environments. What is strange is it mainly appears in a simple query on a ver small table (prod ~20 rows staging ~5 rows) and the query is based on primary key. Our load can be high on prod at times but is extremely low in staging.

query
select * from public.companies where id = ‘GUID’

What is more strange is that it is not a repeatable event. In other words, it only happens sometimes. We have been looking for the issue for about a month now. Any help would be appreciated. It is always this query … very strange not to see it on random queries if it was a connection issue. We have check the Postgres logs and see no errors of note.

Thanks — Nick

Full error

XXXXX.Base.NpgsqlDB In at Npgsql.Internal.NpgsqlReadBuffer.g__EnsureLong|41_0(NpgsqlReadBuffer buffer, Int32 count, Boolean async, Boolean readingNotifications)
at Npgsql.Internal.NpgsqlConnector.g__ReadMessageLong|211_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
at Npgsql.Internal.NpgsqlConnector.g__ReadMessageLong|211_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
at Npgsql.NpgsqlDataReader.NextResult()
at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior)
at Npgsql.NpgsqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader(CommandBehavior behavior)
at System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset, DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior)
at System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior)
at System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, String srcTable)
at CapTrav.Base.NpgsqlDb.Execute_QueryWithOpenConnection(String sqlStr, String tableName, DataSet ds, NpgsqlConnection conn) in C:UsersnromanosourcereposCapTrav_coreCapTrav.BaseNpgsqlDB.cs:line 118 with message = Exception while reading from stream in CapTravel with Inner Exception = Attempted to read past the end of the stream. sent message = select * from public.companies where id = ‘GUID’
Code — We open one connection and use it to fill multiple tables. The first query is the one that fails.

public bool FillAllCompanies(CompanyDS companyDS)
{
NpgsqlConnection conn = NpgsqlDb.GetConnection(true);

        try
        {
            conn.Open();

            NpgsqlDb.Execute_QueryWithOpenConnection(SqlStrCompanies, companyDS.companies.TableName, companyDS, conn);
            NpgsqlDb.Execute_QueryWithOpenConnection(SqlStrDomains, companyDS.domains.TableName, companyDS, conn);
            NpgsqlDb.Execute_QueryWithOpenConnection(SqlStrDomainRulesLiterals, companyDS.domain_rules_literals.TableName, companyDS, conn);
            NpgsqlDb.Execute_QueryWithOpenConnection(SqlStrDomainRules, companyDS.domain_rules.TableName, companyDS, conn);
            return NpgsqlDb.Execute_QueryWithOpenConnection(SqlStrDomainAuths, companyDS.domain_auths.TableName, companyDS, conn);
        }
        catch (Exception ex) { ErrorHelper.WriteException(ex, ClassName); }
        finally
        {
            if (conn.State == ConnectionState.Open) conn.Close();
            conn.Dispose();
            conn = null;
        }
        return false;
    }

Next level of Code

    public static bool Execute_QueryWithOpenConnection(string sqlStr, string tableName, DataSet ds, NpgsqlConnection conn)
    {

        try
        {
            using (NpgsqlDataAdapter npgsqlDataAdapter = new NpgsqlDataAdapter(sqlStr, conn))
            {
                npgsqlDataAdapter.Fill(ds, tableName);
                return true;
            }
        }
        catch (NpgsqlException npge) 
        {
            if (ErrorHelper.WriteException(npge, ClassName, sqlStr))
                WriteDataSetErrors(ds);
        }
        catch (Exception e) 
        {
            if (ErrorHelper.WriteException(e, ClassName, sqlStr))
                WriteDataSetErrors(ds);
        }

        return false;
    }

@nromano32 as always with timeouts, it’s not really impossible for us to help here; Npgsql is only a client library, and if it doesn’t get a response from PG in the allotted time, this exception occurs. Timeouts are frequently the result of networking conditions, or, more rarely, of long-running queries. Turning on keepalives may help, read the docs for more info.

I’m going to go ahead and close this issue as there’s nothing actionable for the Npgsql side.

Thanks for the response! Maybe I put my question in the wrong place. The error is similar however it is attempting to read past the stream.
«Attempted to read past the end of the stream.»

I take that as a response was returned. It is a super simple query and a very small table. What does that mean?

Thanks — Nick

I’d check the logs on the PostgreSQL side, in case there’s useful information there. Otherwise you can open a new issue with the full exception and stack trace, and ideally a runnable program which reproduces the problem.

Something is definitely going on weird with npgsql. A few times a day getting this same error on a query that fetches 20 rows from a small table. Seems to happen more with recent version of the library. Could be my code but thought I would throw it out there as a data point.

I see this in the postgresql logs about the time the timeout starts happening:

unexpected EOF on client connection with an open transaction

Hello Jeff — Yeah I’m still experiencing this issue. I have played with turning on and off connection pooling with not luck. I’m using AWS RDS proxy to manage connections to the database. My code is in an AWS Lambda and spins up 200 runs concurrently (each run in in and out of the database ~20 times). My hunch is during long standing runs the connection was severed by RDS Proxy.

I get about ~ 3-5 per day but the app seems to recover on the next run. Strangely it only happens in this process. I can also confirm your statement that they seem to be increasing with frequency but not dramatically in my case. I’m on Postgres 14.6, npgsql 7.0.1 and .net core 6.0.13.

I see nothing strange in the postgres logs.

Since the update from version 6.0.6 to 7.0.1, I have been seeing the same issue in the Testcontainers PostgreSQL module. I am not aware of this issue happens before. I have downgraded to the previous version and will monitor to see if it happens again.

FYI I’m testing the fix from this issue —> #3559. I’m thinking that was my issue

I am getting the same error as NpgsqlConnection.Open when the code is executed in a Linux container. It connects and then fails somehow to read data from the stream. When I purposefully set the Host to be a non-existing IP address the error was different saying it cannot connect.

{
  "@timestamp": "2023-03-22T14:21:26.0099707+00:00",
  "level": "Fatal",
  "message": "Unexpected failure",
  "exceptions": [
    {
      "Depth": 0,
      "ClassName": "Npgsql.NpgsqlException",
      "Message": "Exception while reading from stream",
      "Source": "Npgsql",
      "StackTraceString": "   at Npgsql.Internal.NpgsqlReadBuffer.<Ensure>g__EnsureLong|41_0(NpgsqlReadBuffer buffer, Int32 count, Boolean async, Boolean readingNotifications)n   at Npgsql.Internal.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|213_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)n   at Npgsql.Internal.NpgsqlConnector.Authenticate(String username, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)n   at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|193_1(NpgsqlConnector conn, SslMode sslMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)n   at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)n   at Npgsql.ConnectorPool.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)n   at Npgsql.ConnectorPool.<Get>g__RentAsync|31_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)n   at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|45_0(Boolean async, CancellationToken cancellationToken)n   at Npgsql.NpgsqlConnection.Open()n   at CecoChat.Npgsql.NpgsqlDbInitializer.Initialize(NpgsqlOptions options, String database, Assembly scriptSource) in /app/CecoChat.Npgsql/NpgsqlDbInitializer.cs:line 24n   at CecoChat.Server.User.HostedServices.InitUsersDb.StartAsync(CancellationToken cancellationToken) in /app/CecoChat.Server.User/HostedServices/InitUsersDb.cs:line 33n   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)n   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)n   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)n   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)n   at CecoChat.Server.EntryPoint.CreateAndRunHost(IHostBuilder hostBuilder, Type loggerContext) in /app/CecoChat.Server/EntryPoint.cs:line 66",
      "RemoteStackTraceString": null,
      "RemoteStackIndex": 0,
      "HResult": -2147467259,
      "HelpURL": null
    },
    {
      "Depth": 1,
      "ClassName": "System.TimeoutException",
      "Message": "Timeout during reading attempt",
      "Source": null,
      "StackTraceString": null,
      "RemoteStackTraceString": null,
      "RemoteStackIndex": 0,
      "HResult": -2146233083,
      "HelpURL": null
    }
  ],
  "fields": {
    "SourceContext": "CecoChat.Server.User.Program",
    "MachineName": "user-75689f6dd8-hphvp",
    "Application": "CecoChat.Server.User"
  }
}

@roji this should get reopened. Hiding behind a timeout error when there are obviously bugs when reading from stream (read the above messages from multiple people) is not right.

EDIT: @roji turned out to be right, read more below.

@roji this should get reopened. Hiding behind a timeout error when there are obviously bugs when reading from stream (read the above messages from multiple people) is not right.

I promise you nobody’s hiding behind anything… But we have yet to receive any bug report we can actually look into. I understand that it’s especially difficult to reproduce this kind of bug, but it’s not impossible, and nobody has done that so far. We haven’t even seen a single packet dump that would indicate an issue on the Npgsql side.

On the other hand, timeouts really do occur because of network devices — I’ve seen that happen many times.

So I can open the issue, but I don’t see what we can do about it until someone provides the relevant info for us to look into…

In my case, it was AWS RDS Proxy resetting the connection as described here #3559

I have not seen this issue since implementing the fix.

@nromano32 yes, I read that but it was not related to me. In my case Connection.Open was returning Exception while reading from stream. I was not even able to connect so that my connection to be reset afterwards 🙂

@roji yes, I was wrong and you were right. I spent hours digging and found out the networking problem. I am using Npgsql to connect to YugabyteDB which exposes PostgreSQL query interface (along with Cassandra and Redis ones, too). Initially I successfully set up a Yugabyte master+tserver via docker compose which worked. Then I moved to Kubernetes to create a cluster there but because of Kubernetes networking it didn’t work. Binding to POD_IP doesn’t work for Yugabyte and essentially incoming connections were failing. After I bound to 0.0.0.0 it worked.

@roji so my feedback regarding Npgsql would be this. If the host is found, but the port is not being listened to, Npgsql should report connection refused. It was, unfortunately, reporting Exception while reading from stream which I think makes sense for me to believe that there is an actual connection to the remote host at the specified port. Is that a fair assessment or am I wrong again 😄? If you want to experiment I can send you the Kubernetes YML files which would require just minikube. The official production-ready Yugabyte Kubernetes support requires tons of stuff on top of the basic Kubernetes setup.

These are the related issues I found helpful during my troubleshooting:

• Problem with port-forward yugabyte/yugabyte-db#4677
• https://forum.yugabyte.com/t/how-to-port-forward-correctly/1181

@cvetomir-todorov the point of Exception while reading from stream exception is to wrap an exception we get from NetworkStream, so connection refused is probably in the inner exception. It might be a good idea for us to change the message to something like Exception while reading from stream. See more details in inner exception.

@vonzshik the internal error was a timeout one, not connection refused. That’s why I got confused. I did k exec -it yugabyte-tserver-0 -- bash and netstat -an showed me that there is a listener on 5433 port.

@cvetomir-todorov that’s definitely curious. Maybe you should raise that with runtime folks, as we just get that exception from NetworkSteam?

@vonzshik ah I see your point — it’s in someone else’s garden. I assume it’s something related to the convoluted Kubernetes networking and/or minikube bug. I would have to create a minimal example to showcase it. Thanks for clearing things up.

@cvetomir-todorov yeah, it’s important to understand that Npgsql just uses standard .NET APIs (e.g. NetworkStream) to do networking; we definitely don’t control which exception gets raised on failure — we just bubble up whatever the .NET APIs throw.

Maybe this may help. I too have this issue and recently instrumented my app with Sentry. It captured two unhandled exceptions fo this error;

/_/src/Npgsql/Internal/NpgsqlReadBuffer.cs

System.IO.EndOfStreamException: Attempted to read past the end of the stream.
File «NpgsqlReadBuffer.cs», line 186, in async void NpgsqlReadBuffer.Ensure(int count)+EnsureLong(?)
Npgsql.NpgsqlException: Exception while reading from stream
File «NpgsqlReadBuffer.cs», line 258, in async void NpgsqlReadBuffer.Ensure(int count)+EnsureLong(?)
File «NpgsqlConnector.cs», line 1301, in async IBackendMessage NpgsqlConnector.ReadMessage(DataRowLoadingMode dataRowLoadingMode)+ReadMessageLong(?) x 2
File «NpgsqlDataReader.cs», line 425, in async Task NpgsqlDataReader.NextResult(bool async, bool isConsuming, CancellationToken cancellationToken)
File «NpgsqlCommand.cs», line 1414, in async ValueTask NpgsqlCommand.ExecuteReader(CommandBehavior behavior, bool async, CancellationToken cancellationToken) x 2
File «NpgsqlCommand.cs», line 1217, in async ValueTask NpgsqlCommand.ExecuteScalar(bool async, CancellationToken cancellationToken)
File «/_/Dapper/SqlMapper.Async.cs», line 1219, col 9, in async Task SqlMapper.ExecuteScalarImplAsync(IDbConnection cnn, CommandDefinition command)

AND:

/_/src/Npgsql/Internal/NpgsqlReadBuffer.cs

System.ObjectDisposedException: The CancellationTokenSource has been disposed.
File «CancellationTokenSource.cs», line 366, in void CancellationTokenSource.CancelAfter(uint millisecondsDelay) x 2
File «ResettableCancellationTokenSource.cs», line 136, in void ResettableCancellationTokenSource.Stop()
File «NpgsqlReadBuffer.cs», line 207, in async void NpgsqlReadBuffer.Ensure(int count)+EnsureLong(?)
File «NpgsqlConnector.cs», line 1301, in async IBackendMessage NpgsqlConnector.ReadMessage(DataRowLoadingMode dataRowLoadingMode)+ReadMessageLong(?) x 2
File «NpgsqlDataReader.cs», line 425, in async Task NpgsqlDataReader.NextResult(bool async, bool isConsuming, CancellationToken cancellationToken)
File «NpgsqlCommand.cs», line 1414, in async ValueTask NpgsqlCommand.ExecuteReader(CommandBehavior behavior, bool async, CancellationToken cancellationToken) x 2
File «NpgsqlCommand.cs», line 1217, in async ValueTask NpgsqlCommand.ExecuteScalar(bool async, CancellationToken cancellationToken)
File «/_/Dapper/SqlMapper.Async.cs», line 1219, col 9, in async Task SqlMapper.ExecuteScalarImplAsync(IDbConnection cnn, CommandDefinition command)

I found this comment in the code along the stack trace;
// Most of the time, it should be fine to reset cancellation token source, so we can use it again
// It’s still possible for cancellation token to cancel between reading and resetting (although highly improbable)
// In this case, we consider it as timed out and fail with OperationCancelledException on next ReadAsync
// Or we consider it not timed out if we have already read everything (count == 0)
// In which case we reinitialize it on the next call to EnsureLong()
if (async && count > 0)
buffer.Cts.RestartTimeoutWithoutReset();

It really seems there is a bug here, likely with the async implementations .

On the surface it looks like a connection timeout issue, it however bubbles up as something that is confusing to everyone (this part seems very likely a bug on npgsql side). Really it should just say timeout so that we investigate other areas.

os: linux
runtime: 6.0.13

Edit: for anyone this may help. For me the issue looks to certainly be timeouts. not caused by a slow query but the cost of opening a new connection ( i was closing aggressively and once i removed that code i am observing less of this). I found for example a span where it took a 1 second to open a connection and 30 milliseconds to execute the query.

@xfiodembo which version of npgsql are you using? If it isn’t the latest 7.0.4, please upgrade to that. Otherwise, can you please open a new issue with the above details?

this is still 6.0.3 , i can give the upgrade a try.

1. Setting the Client Certificate connection string parameter.  
2. Specifying the PGSSLCERT environment variable.
3. Using ~/.postgresql/postgresql.crt (%APPDATA%postgresqlpostgresql.crt on Windows)

Good afternoon Vera.
I passed this information to our architects team, which are the ones in charge of deploying the Flexmonster server, and they haven’t been able to deploy it succesfully.
They tried to run it with ubuntu 20.04 and got the following error message: «No usable version of libssl was found …»
They told me to ask you which version/package of openssl is working/required, and if there is a quicker contact way they can address you directly their issues while deploying it.

 
Thanks for everything and best regards.
Javier

Hello Javier,

 
Thank you for the update.
Actually, we haven’t been reported about «No usable version of libssl was found» error previously, so it seems we need more time to investigate it. We will keep you updated.

 
However, after a brief research, it seems that a possible solution can be installing libopenssl1_0_0 package (more details are here https://stackoverflow.com/questions/53139591/dotnet-new-command-in-opensuse-causes-no-usable-version-of-the-libssl-was-fou ).

 
As for the communication channels, currently, our Forum is the most optimal way because all technical support team members have direct access to it and can respond quickly.

 
Please let us know if it helps.

 
Regards,
Ian

1. .NET Core 3.0 supports OpenSSL 1.1: https://github.com/dotnet/docs/issues/13475
2. If you have LibreSSL instead of OpenSSL: https://github.com/dotnet/runtime/issues/24869

Hello, Javier, 

 
How have you been? 

 
Our team would like to kindly take an interest in whether you found our previous response helpful. 
Did it work to configure the SSL connection?

 
Looking forward to your feedback.

 
Kind regards, 
Vera

Good afternoon Vera.

 
It took us a few days but using all your indications we were able to deploy it. Thanks for your feedback.

 
Best regards.
Javier

Kind regards, 
Vera