Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-28
07:45:47 sahid it seems for me model wil always be VIF_MDEL_VIRTIO that because CONF.libvirt.use_virtio_for_bridges is defaulted to True
07:46:08 sahid so your comment is not really clear, i mean i don't undertand why my initial suggestion is not working
07:47:03 moshele because if the default is CONF.libvirt.use_virtio_for_bridges is True and you will use direct port it will failed
07:47:09 sahid oh i see :) that is the problem actually :) even for VIF DIRECT model is equal to VIRTIO
07:47:54 moshele shaid: and because I don't understandthis option "CONF.libvirt.use_virtio_for_bridges" I didn't want to change the logic there
07:48:12 moshele sahid: ^
07:48:18 sahid yes i can understand you want to limit the change
07:48:26 sahid i m goint to comment on the review
07:48:40 sahid and see if everyone are agree
07:48:56 moshele sahid: we update the macvtap ci to run with the rx/tx quues option
07:49:51 moshele sahid: and the macvtap ci is passing but it only configure the rx_queue_size
07:50:16 moshele sahid: http://13.74.249.42/92/595592/2/check-nova/Nova-MACVTAP-ML2-Sriov/ea9063b/logs/n-cpu.service.log.gz look for rx_queue_size and not the tx_queue_size
07:50:54 moshele sahid: it seem that the tx_queue_size is configured with vshostuser is that correct behaviour?
07:52:28 sahid yes
07:52:54 sahid for vhostuser you should also be able to configure RX
07:53:03 moshele sahid: ok cool
07:54:42 moshele sahid: so my fix will make it work on rx_queue_size macvtap
07:55:28 moshele sahid: I will update my commit soon. thank for the help :)
07:56:16 sahid moshele: thanks for your work on it :)
08:05:05 openstackgerrit Tushar Patil proposed openstack/nova-specs master: Bi-directional enforcement of traits https://review.openstack.org/593475
08:21:55 openstackgerrit Alex Xu proposed openstack/nova-specs master: Resource retrieving: add change-before filter https://review.openstack.org/591976
08:56:08 moshele sahid: another way is that I will change the code to skip setting the model to virtio in https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L137-L142 for direct passthoght ports and then I can do you check of the virtio model. I think this will be cleaner
09:21:44 openstackgerrit Stephen Finucane proposed openstack/nova master: conf: Use new-style choice values https://review.openstack.org/530924
09:24:07 openstackgerrit Moshe Levi proposed openstack/nova master: libvirt: skip setting rx/tx queue sizes for not virto interfaces https://review.openstack.org/595592
09:55:07 sahid moshele: yes sounds, can you comment so other contributors that are involved on the patch can understand why you decided to do that :)
10:46:12 Tahvok Hey guys! I'm trying to understand why nova conductor is receiving a lot of messages.. I mean I get from 30 to 120 messages a second. The environment is not small, we have around 160 compute hosts, but it's not very active. We have a new instance coming up/deleted every hour or so
10:47:08 Tahvok Apart from it, we have our rabbit service working with 100% up to 400% cpu all the time, and along with it, we see the nova-conductor service processes taking 10% cpu each
10:53:12 sean-k-mooney Tahvok: i could be wrong but it think all database acess from the computenodes is relayed via the conductor so the periodic jobs that update the compute node resouces and health will be a portion of those messages
10:53:50 Tahvok sean-k-mooney: thanks.. I
10:54:12 Tahvok I've just tried something else: default_log_levels = oslo,messaging=DEBUG
10:54:35 Tahvok And I see lots of messages like this: 2018-08-28 05:54:26.497 23601 DEBUG oslo.messaging._drivers.impl_rabbit [-] Timed out waiting for RPC response: Timeout while waiting on RPC response - topic: "<unknown>", RPC method: "<unknown>" info: "<unknown>" _raise_timeout /openstack/venvs/nova-15.1.25/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:1052
10:55:16 Tahvok What are this unknown calls? Also, rabbit seems to be working fine, as we don't have any timeout issues when creating new instances
10:55:26 sean-k-mooney hum that not very desciptive. we probaly should have better logging
10:55:49 Tahvok Btw, I'm on Ocata if that matters
10:57:00 sean-k-mooney stephenfin: any RPC people around that you can think of. i would ping dansmith but he should be sleeping for another few hours
10:57:34 Tahvok What I'm trying to fix is basically this periodic (every second or 2) spikes: http://paste.openstack.org/show/728925/
10:58:23 Tahvok Each infra host has 32 thread cores, I think it should be enough to handle 160 compute hosts
10:58:50 sean-k-mooney Tahvok: so every few seconds the condoctor is taking 100% cpu across all cores?
10:59:01 Tahvok I might be wrong thought, and we need to increase our resources, that's why I'm consulting with you
10:59:22 sean-k-mooney Tahvok: do you have 1 controler or several
10:59:39 sean-k-mooney Tahvok: also no you should be fine
11:00:13 Tahvok sean-k-mooney: not all cores, as there are only 18 nova-conductor processes running, at least according to this: http://paste.openstack.org/show/728926/
11:00:43 Tahvok sean-k-mooney: 2 controllers, and we plan to add another one by the end of this week
11:00:44 sean-k-mooney 1 contoler should be eaislly able to handle 160 nodes that are more or less idel in terms of vm lifcyle events
11:02:20 sean-k-mooney Tahvok: my guess is the spikes are caused by the periodic jobs. if all the clocks are synced all 160 nodes will submit the there updates around the same time
11:03:07 sean-k-mooney we proables should be intoducing some spread in when they run. that said without any logs/error that is jsut a guess
11:03:29 Tahvok We have other network issues that network team is handling right now (ksoftirqd is taking lots of cpu (80%+)), but we are trying to fix the rabbit issue. That's what we see on the active router controller: http://paste.openstack.org/show/728928/
11:04:11 Tahvok sean-k-mooney: don't they submit the sync every minute? It doesn't explain ~100 messages every second
11:05:20 Tahvok And we don't have ceilometer, so rabbit handles only the basic openstack services
11:09:23 sean-k-mooney they do. but that does not seam that high to me for a cloud of your size.
11:13:10 Tahvok sean-k-mooney: so 100+ messages a second is normal for this size?
11:18:09 sean-k-mooney Tahvok: i unfortuetly dont have that data to hand.
11:24:32 sean-k-mooney Tahvok: we did some sacle testing back 2 years ago https://review.openstack.org/#/c/352101/ that im quickly checking.
11:25:59 sean-k-mooney Tahvok: that was done with a cloud with 3 contolers and about 230 compute nodes. if i remember correctly the total cpu usage on the contolers was in low 10% for the majority of the testing
12:02:08 stephenfin sean-k-mooney: bauzas would be my other suggestion but he's still on vacation
12:07:51 sean-k-mooney stephenfin: ya he came to mind. Tahvok i would suggest asking again in an hour or so. the us based cores that work on the conductor will be online then and perhaps can give a better answer
12:08:28 Tahvok sean-k-mooney: ok, thanks!
12:42:04 openstackgerrit Konstantinos Samaras-Tsakiris proposed openstack/os-traits master: Add CUDA versions 8 and 9 https://review.openstack.org/597111
12:47:38 openstackgerrit Merged openstack/nova master: Deprecate Core/Ram/DiskFilter https://review.openstack.org/596502
13:12:36 gibi mriedem: hi! Do you have topics for the today's notification subteam meeting?
13:13:17 mriedem nope
13:15:40 gibi mriedem: cool, then I will cancel
13:15:48 mriedem wfm
13:17:56 openstack Launchpad bug 1739325 in OpenStack Compute (nova) ocata "Server operations fail to complete with versioned notifications if payload contains unset non-nullable fields" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
13:17:56 gibi mriedem: we got a report that not just the flavor.disabled is missing in some old embedded falvors but flavor.is_public as well. https://bugs.launchpad.net/nova/+bug/1739325 I will look at it a bit later but I guess a solution will be similar than for flavor.disabled
13:19:23 mriedem gibi: i suppose,
13:19:34 mriedem difference between disabled and is_public is disabled isn't in any API but is_public is
13:20:23 gibi so I have to check that what happens on the API if such old flavor is present and fix possible failures there as well
13:20:46 mriedem well,
13:21:08 mriedem this isn't a failure on flavor resources, or shouldn't be, it's embedded flavors in the instance which were originally migrated from the instance system_metadata
13:21:57 mriedem this is the problem http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/flavors.py#n52
13:22:09 mriedem is_public isn't in there, so it wasn't stored in the embeded instance.flavor
13:22:15 mriedem so we'll have to default to is_public=True
13:22:20 mriedem if it's not in the embedded flavor
13:24:32 mriedem i left a comment on the bug
13:24:40 gibi mriedem: OK, I have to look at the API code showing the embeded flavor to see how is_public is handled there
13:24:50 mriedem i'll report a new bug for is_public since this other one is already closed
13:25:27 mriedem we don't show that field from the instance.flavor
13:25:58 mriedem http://git.openstack.org/cgit/openstack/nova/tree/nova/api/openstack/compute/views/servers.py#n349
13:26:56 gibi mriedem: then the API is not broken. cool. You are way faster to find these things in the code than me.
13:27:07 gibi mriedem: I agree that it needs a separate bug
13:27:24 gibi mriedem: thanks for reporting it
13:27:36 openstack Launchpad bug 1789423 in OpenStack Compute (nova) "Server operations fail to complete with versioned notifications if payload contains unset is_public field" [Undecided,New]
13:27:36 mriedem https://bugs.launchpad.net/nova/+bug/1789423
13:28:17 mriedem lyarwood: speaking of which, can you hit this? https://review.openstack.org/#/c/580525/
13:32:02 lyarwood mriedem: yup looking
13:34:22 gibi jaypipes: thanks for the mail about the consumer gen handling in nova I think efried_doc's and your comment together helps me redo the patch series (now I have to find the time to do it).
13:42:04 jaypipes gibi: np
14:07:24 kosamara Hi efried, are you around?
14:09:59 openstackgerrit Jay Pipes proposed openstack/os-traits master: Add CUDA versions 8 and 9 https://review.openstack.org/597111
14:10:32 jaypipes stephenfin: can you +2/W https://review.openstack.org/#/c/597111/ please?
14:11:11 stephenfin jaypipes: Sure, done
14:11:36 jaypipes stephenfin: danke
14:11:48 jaypipes kosamara: ^^
14:11:54 jaypipes kosamara: thx mate
14:20:26 kosamara jaypipes: thanks!
14:23:48 kosamara efried: I'd like to take up your offer :) From the 4 major things to address on the spec, could you focus on 2,3,4 so that I can focus on 1 for libvirt?
14:24:54 kosamara efried: I'll also see if I can carry over content from your OOT spec for powervm there. And I think alex_xu's questions are part of the "cyborg intersection".
14:43:48 mriedem tommylikehu: you had a question about volumes?
14:44:05 tommylikehu yeah, I have a question regarding deleting operations, When deleting instance and its related volumes, there could be a period of time that those volumes' statues can be available, right? I mean right after the volume is detached

Earlier   Later