| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-22 | |||
| 13:07:14 | sean-k-mooney | sq4ind: did you set allocation in aggregates or on compute node nova.conf | |
| 13:07:23 | sq4ind | on compute | |
| 13:07:37 | sean-k-mooney | sq4ind: oh ok. we broke the aggregate allocation ratios | |
| 13:08:38 | sean-k-mooney | sq4ind: can you share the full resouce provider info for 52c0c39e-30f9-4bd8-84e9-af5c35aac61f | |
| 13:08:53 | sq4ind | it looks like the resources are not being properly updated | |
| 13:09:45 | openstackgerrit | Eric Fried proposed openstack/nova master: Change compute mgr placement check to region_name https://review.openstack.org/554759 | |
| 13:09:59 | sean-k-mooney | sq4ind: well you resocce useage exceed you cappasity currently 175104>122355.0 by a ratio of 1.4 | |
| 13:10:20 | sq4ind | Swap: 3815 0 3815 | |
| 13:10:20 | sq4ind | Mem: 120694 6338 113954 9 402 107727 | |
| 13:10:20 | sq4ind | total used free shared buff/cache available | |
| 13:10:20 | sq4ind | [root@nova0 ~]# free -m | |
| 13:10:20 | sq4ind | sean-k-mooney, but how: | |
| 13:10:21 | sq4ind | ? | |
| 13:10:38 | sean-k-mooney | if the ratio is not set in the RP and is at the default of 1.0 then it would fail with that message | |
| 13:10:48 | sean-k-mooney | free -m show current inuse | |
| 13:10:52 | sean-k-mooney | memory | |
| 13:10:57 | sean-k-mooney | not the reseved memory | |
| 13:11:16 | sean-k-mooney | if you are using kvm it does not preallocate the vm memory and only allocates as guests use it | |
| 13:11:35 | efried | mriedem: I went ahead and made that change -----^ | |
| 13:11:54 | efried | ...and rechecked the devstack side - although there's no way the devstack change fails because of this tweak. | |
| 13:12:04 | sean-k-mooney | sq4ind: what does the hyperviors api say is used on nova0 | |
| 13:12:05 | efried | because now we're both setting and checking the new value. | |
| 13:12:39 | efried | mriedem: I think we'd be looking for the nova patch itself to fail tempest now, because it's using devstack with os_region_name set. | |
| 13:13:07 | mriedem | aye aye | |
| 13:13:17 | sq4ind | sean-k-mooney, | |
| 13:13:19 | sq4ind | | free_disk_gb | 492681 | | |
| 13:13:19 | sq4ind | | disk_available_least | 252971 | | |
| 13:13:19 | sq4ind | | current_workload | 0 | | |
| 13:13:22 | sq4ind | | free_ram_mb | 110067 | | |
| 13:13:26 | sq4ind | | host_ip | 10.252.16.190 | | |
| 13:13:30 | sq4ind | | host_time | 13:12:45 | | |
| 13:13:34 | sq4ind | | hypervisor_hostname | nova0.linguamatics.com | | |
| 13:13:38 | sq4ind | | hypervisor_type | QEMU | | |
| 13:13:44 | sq4ind | | hypervisor_version | 2009000 | | |
| 13:13:48 | sq4ind | | id | 1 | | |
| 13:13:49 | sean-k-mooney | sq4ind: pastbing might be simpler | |
| 13:13:50 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Allow to specify granular CPU feature flags https://review.openstack.org/534384 | |
| 13:13:52 | sq4ind | | load_average | 0.10, 0.05, 0.06 | | |
| 13:13:56 | sq4ind | | local_gb | 492834 | | |
| 13:14:00 | sq4ind | | local_gb_used | 153 | | |
| 13:14:04 | kashyap | sq4ind: Please use pastebin :-( | |
| 13:14:04 | sq4ind | | memory_mb | 122867 | | |
| 13:14:08 | sq4ind | | memory_mb_used | 12800 | | |
| 13:14:14 | sq4ind | | running_vms | 4 | | |
| 13:14:18 | sq4ind | | service_host | nova0.linguamatics.com | | |
| 13:14:22 | sq4ind | | service_id | 12 | | |
| 13:14:26 | sq4ind | | state | up | | |
| 13:14:30 | sq4ind | | status | enabled | | |
| 13:14:34 | sq4ind | | uptime | 1:59 | | |
| 13:14:38 | sq4ind | | users | 1 | | |
| 13:14:44 | sq4ind | | vcpus | 16 | | |
| 13:14:48 | sq4ind | | vcpus_used | 6 | |
| 13:14:50 | sq4ind | sorry | |
| 13:14:52 | sq4ind | sean-k-mooney, sorry for pasting here | |
| 13:14:54 | sq4ind | https://pastebin.com/aprda4We | |
| 13:15:13 | sean-k-mooney | sq4ind: thats ok | |
| 13:15:18 | mriedem | lyarwood: can you check https://review.openstack.org/#/c/555029/ before we do a queens release? | |
| 13:17:31 | mriedem | melwitt: might want to throw the spec review day on the schedule wiki https://wiki.openstack.org/wiki/Nova/Rocky_Release_Schedule#Special_review_days | |
| 13:17:48 | sean-k-mooney | sq4ind: so the capsity on the RP more or less matache the hypervisor api 122355.0 vs 122867 | |
| 13:18:10 | sean-k-mooney | sq4ind: but the used ram is way off 12800 vs 175104 | |
| 13:19:01 | sean-k-mooney | efried: any idea what could cause ^ other then leaked allocations | |
| 13:20:41 | efried | sean-k-mooney: TBH I never understood the allocation ration breakage issue. | |
| 13:20:41 | sean-k-mooney | the difference between 122355.0 vs 122867 is placement is in MiB and hypervior api is in MB i think | |
| 13:20:56 | efried | s/ration/ratio/ | |
| 13:21:32 | sean-k-mooney | efried: if you set them in the host aggregate we did not use that value to set the RP allocation ratio and instead only used the nova conf version | |
| 13:21:44 | sean-k-mooney | at least i think that was the issue | |
| 13:22:06 | efried | In this case it's nowhere close, though, as sq4ind points out. | |
| 13:22:26 | sean-k-mooney | well the capasity is right it the used that is wrong | |
| 13:22:36 | efried | Although in the original statement it's a lot closer: "in the placement-api : Over capacity for MEMORY_MB on resource provider 52c0c39e-30f9-4bd8-84e9-af5c35aac61f. Needed: 2048, Used: 175104, Capacity: 122355.0" | |
| 13:22:41 | sean-k-mooney | so this is noting to do with alocation ratios | |
| 13:23:02 | efried | sq4ind: Are you able to query the placement API directly? | |
| 13:23:13 | sq4ind | efried, let me try | |
| 13:23:32 | efried | I would like to see what placement thinks the allocation ratio is for that MEMORY_MB inventory record on provider 52c0c39e-30f9-4bd8-84e9-af5c35aac61f | |
| 13:23:52 | edleafe | cdent: reading scrollback | |
| 13:23:53 | sean-k-mooney | efried: yes the orignial error is correct allocating an addtion 2048 on top of 175104 would violate the over commit ratio | |
| 13:24:09 | efried | sean-k-mooney: Unless the alloc ratio is 1.5 | |
| 13:24:14 | efried | or higher | |
| 13:24:16 | edleafe | cdent: we should accept either, but normalize how we store them. Is that the change you are proposing? | |
| 13:24:35 | efried | edleafe: It is unclear whether we are normalizing them or not. | |
| 13:24:41 | efried | cdent is finding out. | |
| 13:25:02 | edleafe | efried: gotcha. We definitely *should* be normalizing | |
| 13:25:10 | efried | edleafe: Or only accepting one format. | |
| 13:26:24 | edleafe | no, I don't think we need to do that | |
| 13:28:05 | cdent | here's the bug: https://bugs.launchpad.net/nova/+bug/1758057 | |
| 13:28:06 | openstack | Launchpad bug 1758057 in OpenStack Compute (nova) "When creating uuid-based entities we can duplicate UUIDs" [Undecided,Triaged] | |
| 13:28:15 | cdent | we do not normalize, they are treated as different resource providers | |
| 13:29:48 | sq4ind | efried, sorry but I am not able to query placement api directly | |
| 13:31:00 | cdent | efried, edleafe: gonna have my lunch while that settlles in | |
| 13:32:33 | edleafe | cdent: chew thoroughly! | |
| 13:36:44 | efried | cdent: Okay, so I think we have to fix the bug by normalizing UUIDs for all the APIs, and I don't think we need a microversion for that. Afterwards we can consider whether we want a microversion to further restrict the acceptable input formats. | |
| 13:38:33 | sq4ind | is there any way to repopulate cells in the placement ? (or is it safe to remove cell and recreate it ) | |
| 13:38:39 | bauzas | folks, gentle notice I'm under the water with serious vGPU testing | |
| 13:41:22 | jaypipes | cdent: hey, got your question on UUIDs answered? | |
| 13:42:57 | mriedem | kashyap: how's the libvirt min version bump thing going? | |
| 13:43:04 | kashyap | mriedem: Hi | |
| 13:43:16 | kashyap | mriedem: First fixing the last unit test of this, as we speak: https://review.openstack.org/#/c/534384/4/ | |
| 13:43:28 | kashyap | (Then I'll get to it.) | |
| 13:43:55 | kashyap | Just duking around the last test for the conditional in driver.py. The existing patch & tests all 'pass' | |
| 13:44:11 | kashyap | mriedem: You got a deadline for me? Or was it yesterday? :-) | |
| 13:44:44 | mriedem | would be nice to have that done by milestone 1 in case anything crops up, then we have time later in the release to deal with it | |