| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-27 | |||
| 21:18:55 | mriedem | probably not | |
| 21:19:07 | mriedem | operators will likely go back to self-organizing meetups like they used to | |
| 21:19:15 | mriedem | after we bait and switched them | |
| 21:19:16 | mriedem | suckers! | |
| 21:19:49 | sean-k-mooney | this is why my eye glaze over when i see emails on this topic | |
| 21:22:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add contributor guide for upgrade status checks https://review.openstack.org/596902 | |
| 21:37:25 | melwitt | mgagne_: don't forget to re-propose this spec for stein since we didn't get it merged in time for rocky https://review.openstack.org/580742 | |
| 21:39:03 | openstack | bug 1759420 in OpenStack Compute (nova) "nova does not correctly support HW_DISK_BUS=sata or usb for kvm/qemu" [Medium,In progress] https://launchpad.net/bugs/1759420 - Assigned to sean mooney (sean-k-mooney) | |
| 21:39:03 | openstackgerrit | sean mooney proposed openstack/nova master: add regression test for bug 1759420 https://review.openstack.org/587615 | |
| 21:39:04 | openstackgerrit | sean mooney proposed openstack/nova master: libvirt: fix disk_bus handling for root disk https://review.openstack.org/584999 | |
| 21:44:30 | openstackgerrit | Patricia Domingues proposed openstack/nova master: load up the volume drivers by checking architecture https://review.openstack.org/541393 | |
| 21:46:33 | openstackgerrit | Patricia Domingues proposed openstack/nova master: load up the volume drivers by checking architecture https://review.openstack.org/541393 | |
| #openstack-nova - 2018-08-28 | |||
| 01:32:04 | openstackgerrit | JiaJunsu proposed openstack/nova master: Remove args(os=False) in monkey_patch https://review.openstack.org/568999 | |
| 01:51:04 | openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Make monkey patch work in uWSGI mode https://review.openstack.org/592285 | |
| 01:56:55 | prometheanfire | I should be able to `curl http://169.254.169.254/latest/meta-data/instance-id` even on config-drive instances right? | |
| 02:38:10 | donghm | Hi folks, how can I check this number https://github.com/openstack/nova/blob/master/nova/cmd/status.py#L290 via db query or api? | |
| 02:38:19 | donghm | I'm using nova in master | |
| 02:38:35 | donghm | when I run command: nova-status upgrade check | |
| 02:38:59 | donghm | it return There are no compute resource providers in the Placement service but there are 1 compute nodes in the deployment. | |
| 03:58:21 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Transform libvirt.error notification https://review.openstack.org/484851 | |
| 03:58:39 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Adds view builders for keypairs controller https://review.openstack.org/347289 | |
| 03:58:51 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (3) https://review.openstack.org/574104 | |
| 04:41:12 | crazik | hello | |
| 04:41:30 | crazik | Is it safe to cleanup cell mapping table? | |
| 04:42:05 | crazik | I have almost 10k entries there, looks like it was never purged when instances were deleted... | |
| 04:43:15 | crazik | (I want to delete entries where instance_id is no longer in nova instances table | |
| 04:55:17 | openstackgerrit | Merged openstack/nova master: Make instance_list perform per-cell batching https://review.openstack.org/593131 | |
| 07:26:45 | hongda | dansmith : Excuse me. I am working on this patch: "https://review.openstack.org/#/c/579093/" Do you have any questions about this patch now ? | |
| 07:37:03 | moshele | sahid: hi | |
| 07:37:48 | moshele | sahid: I didn't understand you commnet, do you agree with my change? what are the alternative here? | |
| 07:38:12 | sahid | hello moshele, | |
| 07:38:44 | moshele | sahid: I am talking about this commit https://review.openstack.org/#/c/595592/ | |
| 07:39:07 | sahid | i'm not really agree, so i wanted to know your thinking | |
| 07:39:17 | sahid | because i think if we keep your change like that we will have the same issue at some point for an other vif | |
| 07:40:01 | sahid | OR at least what about to add a comment to indicate the limitation of your fix? | |
| 07:41:26 | moshele | sahid: ok, but I am not sure how to do it otherwise because the network_model.VIF_MODEL_VIRTIO is set anyway by this code https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L137-L142 | |
| 07:41:47 | moshele | sahid: and I didn't find any other way to solve this | |
| 07:42:41 | moshele | sahid: I can add that in the future not virtio nics should be skipped as well | |
| 07:43:04 | moshele | sahid: I mean adding a comment ^ | |
| 07:44:49 | sahid | yes... nut i have a last question | |
| 07:44:51 | sahid | but | |
| 07:45:47 | sahid | it seems for me model wil always be VIF_MDEL_VIRTIO that because CONF.libvirt.use_virtio_for_bridges is defaulted to True | |
| 07:46:08 | sahid | so your comment is not really clear, i mean i don't undertand why my initial suggestion is not working | |
| 07:47:03 | moshele | because if the default is CONF.libvirt.use_virtio_for_bridges is True and you will use direct port it will failed | |
| 07:47:09 | sahid | oh i see :) that is the problem actually :) even for VIF DIRECT model is equal to VIRTIO | |
| 07:47:54 | moshele | shaid: and because I don't understandthis option "CONF.libvirt.use_virtio_for_bridges" I didn't want to change the logic there | |
| 07:48:12 | moshele | sahid: ^ | |
| 07:48:18 | sahid | yes i can understand you want to limit the change | |
| 07:48:26 | sahid | i m goint to comment on the review | |
| 07:48:40 | sahid | and see if everyone are agree | |
| 07:48:56 | moshele | sahid: we update the macvtap ci to run with the rx/tx quues option | |
| 07:49:51 | moshele | sahid: and the macvtap ci is passing but it only configure the rx_queue_size | |
| 07:50:16 | moshele | sahid: http://13.74.249.42/92/595592/2/check-nova/Nova-MACVTAP-ML2-Sriov/ea9063b/logs/n-cpu.service.log.gz look for rx_queue_size and not the tx_queue_size | |
| 07:50:54 | moshele | sahid: it seem that the tx_queue_size is configured with vshostuser is that correct behaviour? | |
| 07:52:28 | sahid | yes | |
| 07:52:54 | sahid | for vhostuser you should also be able to configure RX | |
| 07:53:03 | moshele | sahid: ok cool | |
| 07:54:42 | moshele | sahid: so my fix will make it work on rx_queue_size macvtap | |
| 07:55:28 | moshele | sahid: I will update my commit soon. thank for the help :) | |
| 07:56:16 | sahid | moshele: thanks for your work on it :) | |
| 08:05:05 | openstackgerrit | Tushar Patil proposed openstack/nova-specs master: Bi-directional enforcement of traits https://review.openstack.org/593475 | |
| 08:21:55 | openstackgerrit | Alex Xu proposed openstack/nova-specs master: Resource retrieving: add change-before filter https://review.openstack.org/591976 | |
| 08:56:08 | moshele | sahid: another way is that I will change the code to skip setting the model to virtio in https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L137-L142 for direct passthoght ports and then I can do you check of the virtio model. I think this will be cleaner | |
| 09:21:44 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Use new-style choice values https://review.openstack.org/530924 | |
| 09:24:07 | openstackgerrit | Moshe Levi proposed openstack/nova master: libvirt: skip setting rx/tx queue sizes for not virto interfaces https://review.openstack.org/595592 | |
| 09:55:07 | sahid | moshele: yes sounds, can you comment so other contributors that are involved on the patch can understand why you decided to do that :) | |
| 10:46:12 | Tahvok | Hey guys! I'm trying to understand why nova conductor is receiving a lot of messages.. I mean I get from 30 to 120 messages a second. The environment is not small, we have around 160 compute hosts, but it's not very active. We have a new instance coming up/deleted every hour or so | |
| 10:47:08 | Tahvok | Apart from it, we have our rabbit service working with 100% up to 400% cpu all the time, and along with it, we see the nova-conductor service processes taking 10% cpu each | |
| 10:53:12 | sean-k-mooney | Tahvok: i could be wrong but it think all database acess from the computenodes is relayed via the conductor so the periodic jobs that update the compute node resouces and health will be a portion of those messages | |
| 10:53:50 | Tahvok | sean-k-mooney: thanks.. I | |
| 10:54:12 | Tahvok | I've just tried something else: default_log_levels = oslo,messaging=DEBUG | |
| 10:54:35 | Tahvok | And I see lots of messages like this: 2018-08-28 05:54:26.497 23601 DEBUG oslo.messaging._drivers.impl_rabbit [-] Timed out waiting for RPC response: Timeout while waiting on RPC response - topic: "<unknown>", RPC method: "<unknown>" info: "<unknown>" _raise_timeout /openstack/venvs/nova-15.1.25/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:1052 | |
| 10:55:16 | Tahvok | What are this unknown calls? Also, rabbit seems to be working fine, as we don't have any timeout issues when creating new instances | |
| 10:55:26 | sean-k-mooney | hum that not very desciptive. we probaly should have better logging | |
| 10:55:49 | Tahvok | Btw, I'm on Ocata if that matters | |
| 10:57:00 | sean-k-mooney | stephenfin: any RPC people around that you can think of. i would ping dansmith but he should be sleeping for another few hours | |
| 10:57:34 | Tahvok | What I'm trying to fix is basically this periodic (every second or 2) spikes: http://paste.openstack.org/show/728925/ | |
| 10:58:23 | Tahvok | Each infra host has 32 thread cores, I think it should be enough to handle 160 compute hosts | |
| 10:58:50 | sean-k-mooney | Tahvok: so every few seconds the condoctor is taking 100% cpu across all cores? | |
| 10:59:01 | Tahvok | I might be wrong thought, and we need to increase our resources, that's why I'm consulting with you | |
| 10:59:22 | sean-k-mooney | Tahvok: do you have 1 controler or several | |
| 10:59:39 | sean-k-mooney | Tahvok: also no you should be fine | |
| 11:00:13 | Tahvok | sean-k-mooney: not all cores, as there are only 18 nova-conductor processes running, at least according to this: http://paste.openstack.org/show/728926/ | |
| 11:00:43 | Tahvok | sean-k-mooney: 2 controllers, and we plan to add another one by the end of this week | |
| 11:00:44 | sean-k-mooney | 1 contoler should be eaislly able to handle 160 nodes that are more or less idel in terms of vm lifcyle events | |
| 11:02:20 | sean-k-mooney | Tahvok: my guess is the spikes are caused by the periodic jobs. if all the clocks are synced all 160 nodes will submit the there updates around the same time | |
| 11:03:07 | sean-k-mooney | we proables should be intoducing some spread in when they run. that said without any logs/error that is jsut a guess | |
| 11:03:29 | Tahvok | We have other network issues that network team is handling right now (ksoftirqd is taking lots of cpu (80%+)), but we are trying to fix the rabbit issue. That's what we see on the active router controller: http://paste.openstack.org/show/728928/ | |
| 11:04:11 | Tahvok | sean-k-mooney: don't they submit the sync every minute? It doesn't explain ~100 messages every second | |
| 11:05:20 | Tahvok | And we don't have ceilometer, so rabbit handles only the basic openstack services | |
| 11:09:23 | sean-k-mooney | they do. but that does not seam that high to me for a cloud of your size. | |
| 11:13:10 | Tahvok | sean-k-mooney: so 100+ messages a second is normal for this size? | |
| 11:18:09 | sean-k-mooney | Tahvok: i unfortuetly dont have that data to hand. | |
| 11:24:32 | sean-k-mooney | Tahvok: we did some sacle testing back 2 years ago https://review.openstack.org/#/c/352101/ that im quickly checking. | |
| 11:25:59 | sean-k-mooney | Tahvok: that was done with a cloud with 3 contolers and about 230 compute nodes. if i remember correctly the total cpu usage on the contolers was in low 10% for the majority of the testing | |
| 12:02:08 | stephenfin | sean-k-mooney: bauzas would be my other suggestion but he's still on vacation | |
| 12:07:51 | sean-k-mooney | stephenfin: ya he came to mind. Tahvok i would suggest asking again in an hour or so. the us based cores that work on the conductor will be online then and perhaps can give a better answer | |
| 12:08:28 | Tahvok | sean-k-mooney: ok, thanks! | |
| 12:42:04 | openstackgerrit | Konstantinos Samaras-Tsakiris proposed openstack/os-traits master: Add CUDA versions 8 and 9 https://review.openstack.org/597111 | |
| 12:47:38 | openstackgerrit | Merged openstack/nova master: Deprecate Core/Ram/DiskFilter https://review.openstack.org/596502 | |