Questions tagged [python-s3fs]

For questions related to the Python s3fs library

87 questions More

• Bountied 0
• Unanswered
• Frequent
• Score
• Unanswered (my tags)

How to read partitioned parquet files from S3 using pyarrow in python

I looking for ways to read data from multiple partitioned directories from s3 using python. data_folder/serial_number=1/cur_date=20-12-2012/abcdsd0324324.snappy.parquet data_folder/serial_number=2/...

• python
• parquet
• pyarrow
• fastparquet
• python-s3fs

• 1,708

Overwrite parquet file with pyarrow in S3

I'm trying to overwrite my parquet files with pyarrow that are in S3. I've seen the documentacion and I haven't found anything. Here is my code: from s3fs.core import S3FileSystem import pyarrow as ...

• python
• amazon-s3
• pyarrow
• python-s3fs

• 554

s3fs suddenly stopped working in Google Colab with error "AttributeError: module 'aiobotocore' has no attribute 'AioSession'" [closed]

Yesterday the following cell sequence in Google Colab would work. (I am using colab-env to import environment variables from Google Drive.) This morning, when I run the same code, I get the following ...

• python
• google-colaboratory
• botocore
• python-s3fs

• 645

How to read parquet file from s3 using dask with specific AWS profile

How to read a parquet file on s3 using dask and specific AWS profile (stored in a credentials file). Dask uses s3fs which uses boto. This is what I have tried: >>>import os >>>...

• python
• amazon-s3
• boto3
• dask
• python-s3fs

• 13k

ImportError: Missing optional dependency 'S3Fs'. The S3Fs package is required to handle S3 files. Use pip or conda to install S3Fs

I am using AWS Cloud9 as my IDE. import boto3 import pandas as pd # import s3fs # s3_ob=boto3.resource('s3',aws_access_key_id="xxxxxxxxxx",aws_secret_access_key="xxxxxxxxxxxx") client=boto3.client('...

• python-3.x
• amazon-s3
• pip
• aws-cloud9
• python-s3fs

• 123

S3FS python, credential inline

I am trying to use python s3fs to read files in S3 AWS. I could not find the code to put credential (Access key + Secret) into s3fs code. Can anyone please help me how to set this info along with ...

• python
• amazon-web-services
• python-s3fs

• 6,632

download file using s3fs

I am trying to download a csv file from an s3 bucket using the s3fs library. I have noticed that writing a new csv using pandas has altered data in some way. So I want to download the file directly in ...

• python
• amazon-s3
• python-s3fs

• 710

Pandas read_csv specify AWS Profile

Pandas (v1.0.5) use s3fs library to connect with AWS S3 and read data. By default, s3fs uses the credentials found in ~/.aws/credentials file in default profile. How do I specify which profile should ...

• python
• pandas
• python-s3fs

• 3,814

s3fs gzip compression on pandas dataframe

I'm trying to write a dataframe as a CSV file on S3 by using the s3fs library and pandas. Despite the documentation, I'm afraid the gzip compression parameter it's not working with s3fs. def ...

• python
• amazon-s3
• python-s3fs

• 351

How to mount S3 bucket as local FileSystem?

I have a python app running on a Jupiter-notebook on AWS. I loaded a C-library into my python code which expects a path to a file. I would like to access this file from the S3 bucket. I tried to use ...

• python
• amazon-web-services
• amazon-s3
• python-s3fs

• 1,448

aiobotocore - ImportError: cannot import name 'InvalidIMDSEndpointError'

The code below raise an import exception import s3fs fs = s3fs.S3FileSystem(anon=False) Exception Traceback (most recent call last): File "issue.py", line 1, in <module> import ...

• python-3.x
• boto3
• ubuntu-18.04
• python-s3fs

• 23.2k

Resolving dependencies fails on boto3 and s3fs using poetry

I can install boto3, s3fs and pandas using : pip install boto3 pandas s3fs But it fails with poetry : poetry add boto3 pandas s3fs Here is the error : Because no versions of s3fs match >2023.3.0,&...

• boto3
• python-poetry
• python-s3fs

• 806

cannot import s3fs in pyspark

When i try importing the s3fs library in pyspark using the following code: import s3fs I get the following error: An error was encountered: cannot import name 'maybe_sync' from 'fsspec.asyn' (/usr/...

• apache-spark
• amazon-s3
• pyspark
• filesystems
• python-s3fs

• 1,090

Load CSV file into Pandas from s3 using chunksize

I'm trying to read a very big file from s3 using... import pandas as pd import s3fs df = pd.read_csv('s3://bucket-name/filename', chunksize=100000) But even after giving the chunk size it is taking ...

• python
• pandas
• amazon-s3
• python-s3fs

• 339

pytest How to mock s3fs.S3FileSystem open file

I am trying to mockup the call to open a file in a S3 bucket. The code that I have is: # mymodule.py import s3fs #... def __init__(self): self.s3_filesystem = s3fs.S3FileSystem(anon=False, key=...

• python
• mocking
• pytest
• python-s3fs

• 41

1 2 3 4 5 6 Next