Created: (LB-50) use bulk kafka and cassandra read/insert methods

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Created: (LB-50) use bulk kafka and cassandra read/insert methods

JIRA jira-admin@musicbrainz.org
use bulk kafka and cassandra read/insert methods
------------------------------------------------

                 Key: LB-50
                 URL: http://tickets.musicbrainz.org/browse/LB-50
             Project: ListenBrainz
          Issue Type: Improvement
            Reporter: Alastair Porter


4:35 PM <russss> we're not batching the cassandra writes, nor the kafka reads (if that's possible)
4:35 PM <russss> there is a listenstore.insert_batch(listens) method
4:36 PM <russss> which does scatter-gather on the cassandra insert
4:36 PM <russss> so ideally the process should be bulk-read 1000 listens from kafka (ideally with a timeout), then bulk-insert those into cassandra

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://tickets.musicbrainz.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

_______________________________________________
MusicBrainz-bugs mailing list
[hidden email]
http://lists.musicbrainz.org/mailman/listinfo/musicbrainz-bugs
Reply | Threaded
Open this post in threaded view
|

Updated: (LB-50) use bulk kafka and cassandra read/insert methods

JIRA jira-admin@musicbrainz.org

     [ http://tickets.musicbrainz.org/browse/LB-50?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alastair Porter updated LB-50:
------------------------------

    Component/s: listenstore

> use bulk kafka and cassandra read/insert methods
> ------------------------------------------------
>
>                 Key: LB-50
>                 URL: http://tickets.musicbrainz.org/browse/LB-50
>             Project: ListenBrainz
>          Issue Type: Improvement
>          Components: listenstore
>            Reporter: Alastair Porter
>
> 4:35 PM <russss> we're not batching the cassandra writes, nor the kafka reads (if that's possible)
> 4:35 PM <russss> there is a listenstore.insert_batch(listens) method
> 4:36 PM <russss> which does scatter-gather on the cassandra insert
> 4:36 PM <russss> so ideally the process should be bulk-read 1000 listens from kafka (ideally with a timeout), then bulk-insert those into cassandra

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://tickets.musicbrainz.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

_______________________________________________
MusicBrainz-bugs mailing list
[hidden email]
http://lists.musicbrainz.org/mailman/listinfo/musicbrainz-bugs
Reply | Threaded
Open this post in threaded view
|

Commented: (LB-50) use bulk kafka and cassandra read/insert methods

JIRA jira-admin@musicbrainz.org
In reply to this post by JIRA jira-admin@musicbrainz.org

    [ http://tickets.musicbrainz.org/browse/LB-50?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=37201#action_37201 ]

Alastair Porter commented on LB-50:
-----------------------------------

4:45 PM <russss> ok so regarding the kafka consumer, I think we just need to change it to do listenstore.insert_batch([Listen.from_json(listen) for listen in self.consumer.get_messages(count=1000, block=True, timeout=5)])

> use bulk kafka and cassandra read/insert methods
> ------------------------------------------------
>
>                 Key: LB-50
>                 URL: http://tickets.musicbrainz.org/browse/LB-50
>             Project: ListenBrainz
>          Issue Type: Improvement
>          Components: listenstore
>            Reporter: Alastair Porter
>
> 4:35 PM <russss> we're not batching the cassandra writes, nor the kafka reads (if that's possible)
> 4:35 PM <russss> there is a listenstore.insert_batch(listens) method
> 4:36 PM <russss> which does scatter-gather on the cassandra insert
> 4:36 PM <russss> so ideally the process should be bulk-read 1000 listens from kafka (ideally with a timeout), then bulk-insert those into cassandra

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://tickets.musicbrainz.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

_______________________________________________
MusicBrainz-bugs mailing list
[hidden email]
http://lists.musicbrainz.org/mailman/listinfo/musicbrainz-bugs
Reply | Threaded
Open this post in threaded view
|

Assigned: (LB-50) use bulk kafka and cassandra read/insert methods

JIRA jira-admin@musicbrainz.org
In reply to this post by JIRA jira-admin@musicbrainz.org

     [ http://tickets.musicbrainz.org/browse/LB-50?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Kaye reassigned LB-50:
-----------------------------

    Assignee: Robert Kaye

> use bulk kafka and cassandra read/insert methods
> ------------------------------------------------
>
>                 Key: LB-50
>                 URL: http://tickets.musicbrainz.org/browse/LB-50
>             Project: ListenBrainz
>          Issue Type: Improvement
>          Components: listenstore
>            Reporter: Alastair Porter
>            Assignee: Robert Kaye
>
> 4:35 PM <russss> we're not batching the cassandra writes, nor the kafka reads (if that's possible)
> 4:35 PM <russss> there is a listenstore.insert_batch(listens) method
> 4:36 PM <russss> which does scatter-gather on the cassandra insert
> 4:36 PM <russss> so ideally the process should be bulk-read 1000 listens from kafka (ideally with a timeout), then bulk-insert those into cassandra

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://tickets.musicbrainz.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

_______________________________________________
MusicBrainz-bugs mailing list
[hidden email]
http://lists.musicbrainz.org/mailman/listinfo/musicbrainz-bugs
Reply | Threaded
Open this post in threaded view
|

Closed: (LB-50) use bulk kafka and cassandra read/insert methods

JIRA jira-admin@musicbrainz.org
In reply to this post by JIRA jira-admin@musicbrainz.org

     [ http://tickets.musicbrainz.org/browse/LB-50?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Kaye closed LB-50.
-------------------------

    Resolution: Fixed

See PR38, to be deployed soon.

> use bulk kafka and cassandra read/insert methods
> ------------------------------------------------
>
>                 Key: LB-50
>                 URL: http://tickets.musicbrainz.org/browse/LB-50
>             Project: ListenBrainz
>          Issue Type: Improvement
>          Components: listenstore
>            Reporter: Alastair Porter
>            Assignee: Robert Kaye
>
> 4:35 PM <russss> we're not batching the cassandra writes, nor the kafka reads (if that's possible)
> 4:35 PM <russss> there is a listenstore.insert_batch(listens) method
> 4:36 PM <russss> which does scatter-gather on the cassandra insert
> 4:36 PM <russss> so ideally the process should be bulk-read 1000 listens from kafka (ideally with a timeout), then bulk-insert those into cassandra

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://tickets.musicbrainz.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

_______________________________________________
MusicBrainz-bugs mailing list
[hidden email]
http://lists.musicbrainz.org/mailman/listinfo/musicbrainz-bugs