Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-04-16
15:24:36 efried kashyap: Yes. There's a bp for that, so I'm sure he would welcome the help if you'd like to propose a "fix" for this problem that happens to swap out mox for mock.
15:25:09 efried kashyap: https://review.openstack.org/#/q/project:openstack/nova+branch:master+topic:bp/mox-removal
15:25:29 kashyap efried: I'll see about it. Python unit tests is one my Python weakness area
15:25:47 kashyap First trying to remove all the now-needless compat code due to the version bumps
15:26:10 efried kashyap: No pressure :)
15:27:39 kashyap Thanks for the (non-null) pointer
15:28:28 mdbooth efried kashyap: I'd recommend against mixing those 2 changes, though.
15:28:41 kashyap mdbooth: Yeah, I'm strictly following the "One logical change per patch"
15:28:43 kashyap No madness
15:28:58 kashyap But appreciate the reminder :-)
15:29:28 mdbooth_ kashyap: Don't know if ^^^ sent, but I recommend against mixing those 2 changes.
15:29:53 kashyap mdbooth_: Yep, got sent. And then I wrote: < kashyap> mdbooth: Yeah, I'm strictly following the "One logical change per patch" rule
15:30:06 cfriesen mriedem: M0z122: I think this was the last time I tried to push LVM resize/migration upstream: https://review.openstack.org/#/c/337334/
15:31:53 mdbooth_ cfriesen: Aside: do you get a performance benefit from lvm?
15:32:05 openstackgerrit Merged openstack/nova master: Add config drive link to api-guide https://review.openstack.org/561113
15:32:54 cfriesen mdbooth_: last time we checked, I think there was a difference, but I don't know that we've checked in a while. Have you seen more recent benchmarks?
15:33:23 mdbooth_ cfriesen: No, but I vaguely recall that it was marginal at best.
15:33:32 mdbooth_ cfriesen: Over raw, that is.
15:33:56 mdbooth_ Just wondering if you had data which showed it's worth the bother.
15:34:00 M0z122 cfriesen: many thks will have a look on it
15:35:03 openstackgerrit Artom Lifshitz proposed openstack/nova master: Add DifferentHostFilter to enabled_filters default https://review.openstack.org/561651
15:37:12 mdbooth_ cfriesen: Intuitively, I'd expect that if you fallocate a raw file which isn't fragmented and open it O_DIRECT, all of which I think we do, the performance should be the same as LVM, as the kernel is basically doing the same in both cases.
15:38:41 mdbooth_ cfriesen: Numbers beat intuition of course, though :)
15:52:57 Nisha_Agarwal Hi team, I have 2 stable pike devstack setups- one stacked last week on a freshly imaged system and another stacked around a month ago. When i was trying "nova boot" for ironic nodes on both the setups i land up in this error http://paste.openstack.org/show/719143/ due to which it send the "delete" event to ironic node as soon as it goes to "wait-call-back" state. On debugging we found that it happens because instance's context is set to
15:52:57 Nisha_Agarwal None at some point. We commented out this line https://github.com/openstack/nova/blob/stable/pike/nova/objects/instance.py#L824 and the deploy goes thru.
15:55:13 dansmith melwitt: did you see my comment on that bottom certs patch? I think she reverted the changes she made for you
15:57:07 openstackgerrit Artom Lifshitz proposed openstack/nova master: Add (Same|Different)HostFilter to defaults https://review.openstack.org/561651
15:57:12 Nisha_Agarwal dansmith, could you help me with above? Should i go ahead and open the defect against Nova for the OrphanedObject error reported on stable/pike?
15:57:59 Nisha_Agarwal or there is some configuration in the config file which i should be setting so that the issue is not hit?
16:01:57 dansmith Nisha_Agarwal: orphaned objects are those without a context set, which prevents them from doing things like loading or saving from the database
16:02:04 dansmith Nisha_Agarwal: obviously no config option should cause/prevent that
16:03:46 cfriesen mriedem: we're running into problems with the code that adds a uuid to services on the fly. it seems that calling service.save() on a deleted service doesn't work. it chokes down in service_get()
16:04:04 mriedem cfriesen: where do we save() a deleted service?
16:04:20 cfriesen Service._from_db_object(), if there's no uuid
16:04:35 mriedem might be a bug that melwitt has recently fixed
16:05:32 mriedem cfriesen: https://review.openstack.org/#/c/555923/
16:05:39 mriedem is that it?
16:06:04 dansmith yeah
16:06:06 dansmith was going to say
16:06:53 mriedem if it's not that, then i will need to see a stacktrace rather than a dead cat
16:07:35 openstackgerrit Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949
16:07:36 openstackgerrit Jackie Truong proposed openstack/nova master: Plumb trusted_certs through libvirt driver image paths https://review.openstack.org/561262
16:07:37 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
16:07:39 openstackgerrit Jackie Truong proposed openstack/nova master: Add certificate validation docs https://review.openstack.org/560158
16:07:49 cfriesen mriedem: so previously we were hitting the bug that commit was fixing...so we applied that fix and hit this other problem. Stacktrace at http://paste.openstack.org/show/719308/
16:08:04 Nisha_Agarwal dansmith, the issue is seen on both the stable pike setups as soon as nova calls this https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L1102 which in turn hits https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L452
16:08:16 mriedem gibi: welcome back - you might have an opinion on the last comment i left here about notifications https://review.openstack.org/#/c/486204/
16:08:24 dansmith cfriesen: is that service deleted?
16:08:27 mriedem i'll throw it in the notification meeting
16:08:34 cfriesen dansmith: yes
16:08:51 dansmith cfriesen: that's why, but I thought we determined it shouldn't even be processing deleted services.. did you tweak something in there?
16:09:35 cfriesen dansmith: the sequence of events was: boot instance on compute node, migrate instance, delete compute node and service, upgrade to pike, create new compute node with same name, migrate instance back to that compute node.
16:09:56 cfriesen dansmith: so now when we look up the service by name, there are two matching entries, one of which is deleted
16:10:24 dansmith cfriesen: but we're using model_query which should filter out the deleted one, IIRC
16:10:28 openstackgerrit Jay Pipes proposed openstack/nova-specs master: Numbered request groups use different providers https://review.openstack.org/560974
16:10:32 mriedem cfriesen: i need more of the stacktrace
16:10:37 mriedem not the middle of it
16:11:10 stephenfin bauzas: Want to weigh in on this comment? https://review.openstack.org/#/c/379525/9/nova/scheduler/weights/cpu.py
16:11:17 cfriesen mriedem: will try to get it.
16:11:24 stephenfin bauzas: We don't do this for RAM or disk. Should we?
16:12:36 mriedem stephenfin: the core filter does
16:12:37 mriedem vcpus_total = host_state.vcpus_total * cpu_allocation_ratio
16:13:56 mriedem stephenfin: also, it looks like the HostState object takes into account reserved_host_cpus?
16:14:21 cfriesen mriedem: I think the core/ram/disk filters account for allocation ratio (though disk is wonky), but maybe the weighers don't?
16:15:08 stephenfin mriedem: *don't take into account?
16:15:42 mriedem yeah doens't take into account
16:15:52 mriedem the inventory on the resource provider for the compute node in placement accounts for reserved values
16:16:03 mriedem but i don't see that on the HostState object or it's wrapped compute node
16:16:18 stephenfin mriedem: Line 1136 of nova/compute/resource_tracker.py ?
16:16:24 cfriesen looks like for ram and disk they just weigh them based on the "free" amount
16:16:26 stephenfin (_update_usage_from_instances function)
16:16:50 stephenfin So I'm guessing it gets it from the ComputeNode object
16:17:52 mriedem stephenfin: hmm, yeah, but that's weird.
16:17:56 mriedem cn.vcpus_used = CONF.reserved_host_cpus
16:18:04 mriedem that's not accounting for the number of instances running on the ohst
16:18:05 mriedem *host
16:18:59 stephenfin mriedem: We do that at the end of the function with the call to '_update_usage_from_instance', I suspect
16:19:12 stephenfin Which presumably appends rather than overwrites
16:19:27 stephenfin cfriesen: Hmm, so as we don't have a "free"-style attribute for CPUs I guess they're correct and we've to figure that out ourselves
16:19:34 stephenfin Or add such an attribute
16:20:51 mriedem stephenfin: yeah looks like _update_usage()
16:21:11 mriedem which also takes into account overhead from the virt driver
16:21:11 mriedem vcpus_usage += overhead.get('vcpus', 0)
16:23:22 mriedem ok left some comments in your change for reference
16:23:32 stephenfin mriedem: Excellent. Ta
16:23:35 mriedem i won't profess to know what the right answer is
16:23:46 mriedem it does match this though https://github.com/openstack/nova/blob/fb0b785169e5e422b06e82f2eb58e68f6d2008b3/nova/compute/resource_tracker.py#L786
16:24:46 mriedem this is the claim code
16:24:47 mriedem https://github.com/openstack/nova/blob/fb0b785169e5e422b06e82f2eb58e68f6d2008b3/nova/compute/claims.py#L185
16:24:53 mriedem which is what ultimately matters
16:26:09 mriedem the fun part is,
16:26:27 mriedem if you're using placement, you shouldn't enable the CoreFilter, which means the vcpus limit isn't passed from scheduler to compute, and no vcpus claim is actually done in the resource tracker
16:26:44 cfriesen mriedem: here's the full trace: http://paste.openstack.org/show/719309/
16:26:48 mriedem the only actual RT claim we should ever have, if you're using placement, right now is for numa
16:26:53 mriedem and maybe pci?
16:27:29 mriedem cfriesen: that's the exact same thing that melwitt fixed
16:27:54 cfriesen mriedem: no, melwitt's change was a reader->writer context thing
16:29:24 mriedem hmm, so more similar to Ic2f239f634f917a5771b0401a5073546c710c036
16:29:30 mriedem except in that case the instance is deleted but the service isn't
16:30:36 mriedem dansmith: didn't you recently have a read_deleted thing in the Instance object for evacuate?

Earlier   Later