# Data Export to Cloud Storage (BigQuery / Amazon S3)

devtodev has an option to export user and event data to a cloud storage. The event data is uploaded once every hour. User data is uploaded every day, if we receive at least one event for the last 24 hours.

To export your data to one of the supported cloud storages, please send a request to <info@devtodev.com>.

## Export to BigQuery

To export data to BigQuery you will need to:

1. Create a service account, if it does not already exist.
2. Get service account credentials.
3. Create a dataset.
4. Choose what kind of data you want to export to your dataset.

In the request specify the following details:&#x20;

1. Service account credentials;
2. Name and location of the dataset in BigQuery;&#x20;
3. Export configuration ([see below](https://docs.devtodev.com/integration/data-export/data-export-to-cloud-storage-bigquery-amazon-s3#export-configuration)).

### Creating a service account in BigQuery

1. If you do not have a service account, create one by following the [Google Cloud manual](https://cloud.google.com/iam/docs/creating-managing-service-accounts#creating).
2. Your service account has to have rights for table creation and data upload. Add **bigquery.user** or **bigquery.admin** and one of the following roles to your service account:
   * bigquery.dataEditor
   * bigquery.dataOwner
   * bigquery.jobs.create
   * bigquery.tables.create
3. Create credentials for your service account, if there are none yet. Follow [this manual](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) to create access keys. To create keys, add **serviceAccountKeyAdmin** role to your service account.

### Creating a dataset

Follow [this manual](https://cloud.google.com/bigquery/docs/datasets) to create a dataset in BigQuery.

Name your dataset **devtodev**, that way we can send your data to BigQuery.&#x20;

Also, while creating a dataset, keep **location** in mind.&#x20;

{% hint style="danger" %}
**You cannot change the location of the dataset later!** [More on locations in BigQuery](https://cloud.google.com/bigquery/docs/locations).
{% endhint %}

### Export configuration

After creating a service account and a dataset we need to configure export in devtodev.

You can choose one of two ways to export your data:

**Export data to one table** — all event data will be uploaded to one common table named **p\<project id>\_events**. &#x20;

**Export data by event type** — every event type will be uploaded to their respective table. *The list of event types is below.*&#x20;

Every event type will have a table with a name like this **p\<project id>\_events\_\<event type>\[**\_\<event name>**]**. For example:

* **p234\_rp** —this is a table for **real payment** events from a project with id 234.
* **p234\_ce\_mission\_start** — this is a table for a **custom event** named “mission\_start“  from a project with id 234.

You can match project name and project id in the **\_projects** table, which will be automatically filled at the time of the first export.

Active user information will be uploaded to a separate table named **p\<project id>\_users** regardless of how you choose to upload event data.

## Export to Amazon S3

To export data to Amazon S3 you will need to:

1. Create an account, if it does not already exist.
2. Get credentials (accessKey and secretKey).
3. Create a bucket.
4. Choose what kind of data you want to export to your bucket.

In the request specify the following details:&#x20;

1. Account credentials;
2. Name and region of the bucket in Amazon S3;&#x20;
3. Export configuration ([see below](https://docs.devtodev.com/integration/data-export/data-export-to-cloud-storage-bigquery-amazon-s3#export-configuration-1)).

### Creating an Amazon S3 account

If you do not already have an account, follow [this AWS manual](https://docs.aws.amazon.com/AmazonS3/latest/userguide/setting-up-s3.html) to create one.

### Getting credentials

See [this manual](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html#cli-configure-files-where) for more detail on how to find your credentials.

We need **accessKey** and **secretKey** which are located in \~/.aws/credentials file. We will also need your **region** information, it is located in \~/.aws/config file. Execute [aws configure](https://docs.aws.amazon.com/cli/latest/reference/configure/index.html) command in AWS developer console to get **accessKey** and **secretKey**.

Example:

```
$ aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: json
```

### **Creating a bucket**

Follow [this manual](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) to create a bucket in S3.

The Name of the bucket should be unique, see [more on bucket naming](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html). Also, while creating a dataset, keep the **region** in mind.&#x20;

{% hint style="warning" %}
Objects can never leave the region unless they are explicitly transferred! [More on AWS regions](https://docs.aws.amazon.com/general/latest/gr/rande.html).
{% endhint %}

### Export configuration

After creating an account and a bucket we need to configure export in devtodev.

Your data will be stored in a bucket directory named **p\<project id>** which will store .csv files compressed with gzip. Each directory will have a **project\_info.txt** file with the **project name** and **application id** in devtodev service.

You can choose one of two ways to export your data:

**Export data to one table** — all event data will be uploaded to one **common** table.&#x20;

Example of such table: 2021\_05\_26\_08\_00\_54**common**86ddf8a5-1e7f-4f2c-a4d3-22f4d6a8860c

**Export data by event type** — every event type will be uploaded to their respective table. *The list of event types is below.*

Some examples:

* 2021\_05\_26\_08\_08\_28**ce**\[editor\_item\_remove]a9413576-0a32-4a2a-ad84-940150e9a218 — this is a table for a **custom event** named “editor\_item\_remove“.&#x20;
* 2021\_05\_26\_08\_08\_11**rp**556dbd8d-71c9-41b4-9564-d43b39ca1b7d — this is a table for **real payment** events.

Active user information will be uploaded to a separate users table regardless of how you choose to upload event data.&#x20;

Example of such table: 2021\_05\_26\_08\_07\_52**users**439c129f-d70b-4f98-ad86-4cb01054732b

## **List of event types**

During export configuration you can select what type of events you want to export. You can also select which project should be exported and which should not.

The list below contains event types (with fields) available for export.

### Common basic fields for all event types

```
devtodev_id — numeric user id
main_id — string user id
crossplatform_id — cross platform user id, only for projects with set user id identification
uc_createtime — user registration date
uc_first_paymenttime — first payment date
uc_last_paymenttime — last payment date
uc_payment_cnt — number of payments
uc_payment_sum — sum of payments
uc_level — user level
uc_country — country
uc_language — language
uc_age — age
uc_gender — gender
uc_cheater — mark a cheater
uc_tester — mark a tester
```

### Event types

<table data-header-hidden><thead><tr><th width="228">Event name</th><th width="77.33333333333331">Event code</th><th>Additional fields</th></tr></thead><tbody><tr><td>Event name</td><td>Event code</td><td>Additional fields</td></tr><tr><td>EventTrackingStatus</td><td>ts</td><td>allow_tracking — is tracking allowed</td></tr><tr><td>EventUserInfo</td><td>ui</td><td><p>language — device locale</p><p>custom_udid — custom user id</p></td></tr><tr><td>EventDeviceInfo</td><td>di</td><td><p>device_version</p><p>device_os</p><p>display_resolution</p><p>display_dpi</p><p>androidid</p><p>idfa</p><p>idfv</p><p>advertisingid</p><p>serialid</p><p>manufacturer</p><p>model</p><p>device_model</p><p>offset — user timezone offset<br></p></td></tr><tr><td>EventDeviceInfoV2</td><td>di</td><td><p>device_version
</p><p>device_os
</p><p>display_resolution
</p><p>display_dpi
</p><p>display_diagonal
</p><p>manufacturer
</p><p>model
</p><p>offset — user timezone offset
</p><p>
</p><p>androidid
</p><p>openudid

</p><p>idfa
</p><p>idfv

</p><p>advertisingid
</p><p>serialid
</p><p>install_source
</p><p>user_agent</p><p>
</p></td></tr><tr><td>EventRealPaymentEntry</td><td>rp</td><td><p>currency
</p><p>product
</p><p>payment_id
</p><p>price_usd
</p><p>payment_status
</p></td></tr><tr><td>EventGamePurchase</td><td>ip</td><td><p>amount
</p><p>item_type
</p><p>item
</p><p>inapp_currencies — structure with info on currency type and its amount spent on item purchase</p></td></tr><tr><td>EventCustomColumnar</td><td>ce</td><td><p>event_name
</p><p>event_params — structure with parameter names and values</p></td></tr><tr><td>EventProgression</td><td>pe</td><td><p>location
</p><p>spent
</p><p>earned
</p><p>source
</p><p>difficulty
</p><p>success
</p><p>duration
</p></td></tr><tr><td>EventTester</td><td>tstr</td><td>tester</td></tr><tr><td>EventCheater</td><td>ch</td><td>cheater</td></tr><tr><td>EventRegistrations</td><td>rg</td><td>this event only has basic fields
</td></tr><tr><td>EventGender</td><td>gr</td><td>gender</td></tr><tr><td>EventAge</td><td>ag</td><td>age</td></tr><tr><td>EventGameSessionStart</td><td>ss</td><td>amount</td></tr><tr><td>EventUserEngagement</td><td>ue</td><td>duration</td></tr><tr><td>EventPeople</td><td>pl</td><td><p>name
</p><p>email
</p><p>phone
</p><p>photo
</p><p>event_params — other custom fields</p></td></tr><tr><td>EventSocialNetworkPost</td><td>sp</td><td>network<br>reason</td></tr><tr><td>EventSocialNetworkConnect</td><td>sc</td><td>network</td></tr><tr><td>EventTutorial</td><td>tr</td><td><p>
</p><p>step
</p></td></tr><tr><td>EventLevelUp</td><td>lu</td><td><p>local_duration
</p><p>absolut_duration
</p><p>spent
</p><p>earned
</p><p>balance
</p><p>bought
</p><p>
</p></td></tr><tr><td>EventApplicationInfo</td><td>ai</td><td><p>sdk_version
</p><p>app_version
</p><p>bundle_id
</p><p>engine
</p><p>
</p><p>
</p></td></tr><tr><td>EventWipe</td><td>wipe</td><td><p>save_cheater_tester
</p><p>save_custom_props
</p><p>save_paying_status
</p><p>save_registration
</p></td></tr><tr><td>EventAlive</td><td>al</td><td>this event only has basic fields
</td></tr><tr><td>EventReferal</td><td>rf</td><td><p>publisher
</p><p>sub_publisher
</p><p>sub_ad
</p><p>sub_ad_group
</p><p>sub_campaign
</p><p>sub_placement
</p><p>sub_site
</p><p>cost</p><p>
</p></td></tr><tr><td>EventSubscription</td><td>sbs</td><td><p>source<br>payment_type<br>start_time<br>expiry_time<br>event_type<br>price_usd<br>product<br>original_payment_id </p><p>payment_id<br>purchase_type<br>promo_code<br>promo_type<br>payment_status<br>eventlevel</p></td></tr><tr><td>EventAdRevenue</td><td>adrv</td><td><p>source<br>ad_unit<br>ad_network </p><p>placement </p><p>revenue</p></td></tr></tbody></table>