| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-28 | |||
| 04:41:30 | crazik | Is it safe to cleanup cell mapping table? | |
| 04:42:05 | crazik | I have almost 10k entries there, looks like it was never purged when instances were deleted... | |
| 04:43:15 | crazik | (I want to delete entries where instance_id is no longer in nova instances table | |
| 04:55:17 | openstackgerrit | Merged openstack/nova master: Make instance_list perform per-cell batching https://review.openstack.org/593131 | |
| 07:26:45 | hongda | dansmith : Excuse me. I am working on this patch: "https://review.openstack.org/#/c/579093/" Do you have any questions about this patch now ? | |
| 07:37:03 | moshele | sahid: hi | |
| 07:37:48 | moshele | sahid: I didn't understand you commnet, do you agree with my change? what are the alternative here? | |
| 07:38:12 | sahid | hello moshele, | |
| 07:38:44 | moshele | sahid: I am talking about this commit https://review.openstack.org/#/c/595592/ | |
| 07:39:07 | sahid | i'm not really agree, so i wanted to know your thinking | |
| 07:39:17 | sahid | because i think if we keep your change like that we will have the same issue at some point for an other vif | |
| 07:40:01 | sahid | OR at least what about to add a comment to indicate the limitation of your fix? | |
| 07:41:26 | moshele | sahid: ok, but I am not sure how to do it otherwise because the network_model.VIF_MODEL_VIRTIO is set anyway by this code https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L137-L142 | |
| 07:41:47 | moshele | sahid: and I didn't find any other way to solve this | |
| 07:42:41 | moshele | sahid: I can add that in the future not virtio nics should be skipped as well | |
| 07:43:04 | moshele | sahid: I mean adding a comment ^ | |
| 07:44:49 | sahid | yes... nut i have a last question | |
| 07:44:51 | sahid | but | |
| 07:45:47 | sahid | it seems for me model wil always be VIF_MDEL_VIRTIO that because CONF.libvirt.use_virtio_for_bridges is defaulted to True | |
| 07:46:08 | sahid | so your comment is not really clear, i mean i don't undertand why my initial suggestion is not working | |
| 07:47:03 | moshele | because if the default is CONF.libvirt.use_virtio_for_bridges is True and you will use direct port it will failed | |
| 07:47:09 | sahid | oh i see :) that is the problem actually :) even for VIF DIRECT model is equal to VIRTIO | |
| 07:47:54 | moshele | shaid: and because I don't understandthis option "CONF.libvirt.use_virtio_for_bridges" I didn't want to change the logic there | |
| 07:48:12 | moshele | sahid: ^ | |
| 07:48:18 | sahid | yes i can understand you want to limit the change | |
| 07:48:26 | sahid | i m goint to comment on the review | |
| 07:48:40 | sahid | and see if everyone are agree | |
| 07:48:56 | moshele | sahid: we update the macvtap ci to run with the rx/tx quues option | |
| 07:49:51 | moshele | sahid: and the macvtap ci is passing but it only configure the rx_queue_size | |
| 07:50:16 | moshele | sahid: http://13.74.249.42/92/595592/2/check-nova/Nova-MACVTAP-ML2-Sriov/ea9063b/logs/n-cpu.service.log.gz look for rx_queue_size and not the tx_queue_size | |
| 07:50:54 | moshele | sahid: it seem that the tx_queue_size is configured with vshostuser is that correct behaviour? | |
| 07:52:28 | sahid | yes | |
| 07:52:54 | sahid | for vhostuser you should also be able to configure RX | |
| 07:53:03 | moshele | sahid: ok cool | |
| 07:54:42 | moshele | sahid: so my fix will make it work on rx_queue_size macvtap | |
| 07:55:28 | moshele | sahid: I will update my commit soon. thank for the help :) | |
| 07:56:16 | sahid | moshele: thanks for your work on it :) | |
| 08:05:05 | openstackgerrit | Tushar Patil proposed openstack/nova-specs master: Bi-directional enforcement of traits https://review.openstack.org/593475 | |
| 08:21:55 | openstackgerrit | Alex Xu proposed openstack/nova-specs master: Resource retrieving: add change-before filter https://review.openstack.org/591976 | |
| 08:56:08 | moshele | sahid: another way is that I will change the code to skip setting the model to virtio in https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L137-L142 for direct passthoght ports and then I can do you check of the virtio model. I think this will be cleaner | |
| 09:21:44 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Use new-style choice values https://review.openstack.org/530924 | |
| 09:24:07 | openstackgerrit | Moshe Levi proposed openstack/nova master: libvirt: skip setting rx/tx queue sizes for not virto interfaces https://review.openstack.org/595592 | |
| 09:55:07 | sahid | moshele: yes sounds, can you comment so other contributors that are involved on the patch can understand why you decided to do that :) | |
| 10:46:12 | Tahvok | Hey guys! I'm trying to understand why nova conductor is receiving a lot of messages.. I mean I get from 30 to 120 messages a second. The environment is not small, we have around 160 compute hosts, but it's not very active. We have a new instance coming up/deleted every hour or so | |
| 10:47:08 | Tahvok | Apart from it, we have our rabbit service working with 100% up to 400% cpu all the time, and along with it, we see the nova-conductor service processes taking 10% cpu each | |
| 10:53:12 | sean-k-mooney | Tahvok: i could be wrong but it think all database acess from the computenodes is relayed via the conductor so the periodic jobs that update the compute node resouces and health will be a portion of those messages | |
| 10:53:50 | Tahvok | sean-k-mooney: thanks.. I | |
| 10:54:12 | Tahvok | I've just tried something else: default_log_levels = oslo,messaging=DEBUG | |
| 10:54:35 | Tahvok | And I see lots of messages like this: 2018-08-28 05:54:26.497 23601 DEBUG oslo.messaging._drivers.impl_rabbit [-] Timed out waiting for RPC response: Timeout while waiting on RPC response - topic: "<unknown>", RPC method: "<unknown>" info: "<unknown>" _raise_timeout /openstack/venvs/nova-15.1.25/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:1052 | |
| 10:55:16 | Tahvok | What are this unknown calls? Also, rabbit seems to be working fine, as we don't have any timeout issues when creating new instances | |
| 10:55:26 | sean-k-mooney | hum that not very desciptive. we probaly should have better logging | |
| 10:55:49 | Tahvok | Btw, I'm on Ocata if that matters | |
| 10:57:00 | sean-k-mooney | stephenfin: any RPC people around that you can think of. i would ping dansmith but he should be sleeping for another few hours | |
| 10:57:34 | Tahvok | What I'm trying to fix is basically this periodic (every second or 2) spikes: http://paste.openstack.org/show/728925/ | |
| 10:58:23 | Tahvok | Each infra host has 32 thread cores, I think it should be enough to handle 160 compute hosts | |
| 10:58:50 | sean-k-mooney | Tahvok: so every few seconds the condoctor is taking 100% cpu across all cores? | |
| 10:59:01 | Tahvok | I might be wrong thought, and we need to increase our resources, that's why I'm consulting with you | |
| 10:59:22 | sean-k-mooney | Tahvok: do you have 1 controler or several | |
| 10:59:39 | sean-k-mooney | Tahvok: also no you should be fine | |
| 11:00:13 | Tahvok | sean-k-mooney: not all cores, as there are only 18 nova-conductor processes running, at least according to this: http://paste.openstack.org/show/728926/ | |
| 11:00:43 | Tahvok | sean-k-mooney: 2 controllers, and we plan to add another one by the end of this week | |
| 11:00:44 | sean-k-mooney | 1 contoler should be eaislly able to handle 160 nodes that are more or less idel in terms of vm lifcyle events | |
| 11:02:20 | sean-k-mooney | Tahvok: my guess is the spikes are caused by the periodic jobs. if all the clocks are synced all 160 nodes will submit the there updates around the same time | |
| 11:03:07 | sean-k-mooney | we proables should be intoducing some spread in when they run. that said without any logs/error that is jsut a guess | |
| 11:03:29 | Tahvok | We have other network issues that network team is handling right now (ksoftirqd is taking lots of cpu (80%+)), but we are trying to fix the rabbit issue. That's what we see on the active router controller: http://paste.openstack.org/show/728928/ | |
| 11:04:11 | Tahvok | sean-k-mooney: don't they submit the sync every minute? It doesn't explain ~100 messages every second | |
| 11:05:20 | Tahvok | And we don't have ceilometer, so rabbit handles only the basic openstack services | |
| 11:09:23 | sean-k-mooney | they do. but that does not seam that high to me for a cloud of your size. | |
| 11:13:10 | Tahvok | sean-k-mooney: so 100+ messages a second is normal for this size? | |
| 11:18:09 | sean-k-mooney | Tahvok: i unfortuetly dont have that data to hand. | |
| 11:24:32 | sean-k-mooney | Tahvok: we did some sacle testing back 2 years ago https://review.openstack.org/#/c/352101/ that im quickly checking. | |
| 11:25:59 | sean-k-mooney | Tahvok: that was done with a cloud with 3 contolers and about 230 compute nodes. if i remember correctly the total cpu usage on the contolers was in low 10% for the majority of the testing | |
| 12:02:08 | stephenfin | sean-k-mooney: bauzas would be my other suggestion but he's still on vacation | |
| 12:07:51 | sean-k-mooney | stephenfin: ya he came to mind. Tahvok i would suggest asking again in an hour or so. the us based cores that work on the conductor will be online then and perhaps can give a better answer | |
| 12:08:28 | Tahvok | sean-k-mooney: ok, thanks! | |
| 12:42:04 | openstackgerrit | Konstantinos Samaras-Tsakiris proposed openstack/os-traits master: Add CUDA versions 8 and 9 https://review.openstack.org/597111 | |
| 12:47:38 | openstackgerrit | Merged openstack/nova master: Deprecate Core/Ram/DiskFilter https://review.openstack.org/596502 | |
| 13:12:36 | gibi | mriedem: hi! Do you have topics for the today's notification subteam meeting? | |
| 13:13:17 | mriedem | nope | |
| 13:15:40 | gibi | mriedem: cool, then I will cancel | |
| 13:15:48 | mriedem | wfm | |
| 13:17:56 | openstack | Launchpad bug 1739325 in OpenStack Compute (nova) ocata "Server operations fail to complete with versioned notifications if payload contains unset non-nullable fields" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 13:17:56 | gibi | mriedem: we got a report that not just the flavor.disabled is missing in some old embedded falvors but flavor.is_public as well. https://bugs.launchpad.net/nova/+bug/1739325 I will look at it a bit later but I guess a solution will be similar than for flavor.disabled | |
| 13:19:23 | mriedem | gibi: i suppose, | |
| 13:19:34 | mriedem | difference between disabled and is_public is disabled isn't in any API but is_public is | |
| 13:20:23 | gibi | so I have to check that what happens on the API if such old flavor is present and fix possible failures there as well | |
| 13:20:46 | mriedem | well, | |
| 13:21:08 | mriedem | this isn't a failure on flavor resources, or shouldn't be, it's embedded flavors in the instance which were originally migrated from the instance system_metadata | |
| 13:21:57 | mriedem | this is the problem http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/flavors.py#n52 | |
| 13:22:09 | mriedem | is_public isn't in there, so it wasn't stored in the embeded instance.flavor | |
| 13:22:15 | mriedem | so we'll have to default to is_public=True | |
| 13:22:20 | mriedem | if it's not in the embedded flavor | |
| 13:24:32 | mriedem | i left a comment on the bug | |
| 13:24:40 | gibi | mriedem: OK, I have to look at the API code showing the embeded flavor to see how is_public is handled there | |
| 13:24:50 | mriedem | i'll report a new bug for is_public since this other one is already closed | |
| 13:25:27 | mriedem | we don't show that field from the instance.flavor | |
| 13:25:58 | mriedem | http://git.openstack.org/cgit/openstack/nova/tree/nova/api/openstack/compute/views/servers.py#n349 | |
| 13:26:56 | gibi | mriedem: then the API is not broken. cool. You are way faster to find these things in the code than me. | |
| 13:27:07 | gibi | mriedem: I agree that it needs a separate bug | |
| 13:27:24 | gibi | mriedem: thanks for reporting it | |