| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-22 | |||
| 14:29:39 | mriedem | dansmith: i've abandoned my attempt at fixing https://bugs.launchpad.net/nova/+bug/1815082 but you might be interested in that | |
| 14:29:39 | openstack | Launchpad bug 1815082 in OpenStack Compute (nova) rocky ""DBNonExistentTable: (sqlite3.OperationalError) no such table: services" when starting nova-metadata under uwsgi" [Medium,Triaged] | |
| 14:29:39 | sean-k-mooney | mdbooth: i guess it default to false https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L284 | |
| 14:30:01 | mriedem | dansmith: tl;dr trying to get the indirection api working in n-api-meta in certain cases if we don't have direct access to the db | |
| 14:30:30 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Fix policy doc for host_status and extended servers attribute https://review.opendev.org/689833 | |
| 14:30:55 | mriedem | er i guess we use the indirection api for n-api-meta when running with evetlet but not wsgi | |
| 14:32:31 | dansmith | mriedem: hmm, what you have seems like it should work, although your comment about it hitting superconductor isn't right, unless nova-cpu.conf has api db creds | |
| 14:33:25 | dansmith | mriedem: I could pick up on that, but unless someone is beating down the door for it and/or going to test it prior to merge, I'd probably set it aside too | |
| 14:33:38 | efried | mriedem: I can do those backports, yeah. | |
| 14:33:41 | sean-k-mooney | mriedem: before i go refactor my own work you would prefer i use ProviderUsageBaseTestCase instaead of _IntegratedTestBase and make it non libvirt specific | |
| 14:33:59 | mriedem | jroll: i have given up on https://review.opendev.org/#/c/637217/ - someone else that cares about completing the ironic flavor data migration added in pike is going to have to take over | |
| 14:34:27 | mriedem | dansmith: i had a tempest patch that depended on the nova fix to test it | |
| 14:34:44 | mriedem | https://review.opendev.org/#/q/status:abandoned+topic:bug/1815082 | |
| 14:35:29 | mriedem | dansmith: looking at latest comments the issue was n-api-meta was starting before conductor which blew up | |
| 14:36:21 | mriedem | frickler: ^ i've abandoned those changes for your bug fyi | |
| 14:36:40 | mriedem | sean-k-mooney: yes don't use _IntegratedTestBase for new tests | |
| 14:36:54 | sean-k-mooney | ok thanks | |
| 14:37:15 | mriedem | and yes it'd be cool if things that aren't libvirt specific didn't build on libvirt specific functional test base classes | |
| 14:37:43 | sean-k-mooney | ya i think i can create a small numa file that is not in the libvirt folder to test this | |
| 14:38:06 | sean-k-mooney | although there are few non libvirt numa things that will likely be added to it | |
| 14:39:33 | jroll | mriedem: thanks for the heads up | |
| 14:41:22 | mriedem | sean-k-mooney: just my opinion, don't bend over backward based on something i'm not actively reviewing | |
| 14:46:15 | sean-k-mooney | mriedem: ok be even so i tend to give you opipion weight even if you are not activly reviewing it since others often agree | |
| 14:49:39 | mdbooth | Whoa. It appears from tracing in oslo.concurrency that something is allowing a threading.Semaphore to be taken twice | |
| 14:49:56 | mdbooth | I wonder... is threading.Semaphore re-entrant by default? | |
| 14:50:45 | cdent | those have to be some most mdboothy statements I've ever seen | |
| 14:50:54 | cdent | "hey, everything might be wrong!" | |
| 14:51:12 | cdent | which I, of course, love | |
| 14:51:25 | mdbooth | cdent: I usually hope to follow them up by being wrong, but I like to think them first :) | |
| 14:51:33 | mdbooth | And unfortunately I'm not always wrong. | |
| 14:53:40 | bnemec | It seems highly unlikely that a concurrency class would be non-reentrant. | |
| 14:53:57 | mdbooth | bnemec: s/non-// | |
| 14:54:17 | mdbooth | bnemec: I would *not* expect Semaphore to be re-entrant, and certainly in a regular environment it isn't: | |
| 14:54:35 | bnemec | Are you sure the tracing is in the right place? We spent a bunch of time tracking down a double-locking problem at one point that was caused by logging in the wrong place. | |
| 14:54:48 | mdbooth | >>> import threading >>> s = threading.Semaphore() | |
| 14:54:49 | mdbooth | >>> with s: | |
| 14:54:49 | mdbooth | ... with s: ... print("Hello world") | |
| 14:54:49 | mdbooth | ... | |
| 14:54:51 | mdbooth | Eurgh | |
| 14:54:52 | mdbooth | Sorry | |
| 14:55:02 | mdbooth | Anyway, takes same semaphore twice, hangs | |
| 14:55:40 | bnemec | Oh, well no that wouldn't work. But that's not what reentrancy is. | |
| 14:55:51 | mdbooth | bnemec: Wouldn't rule it out, but I don't think so. It's in a functional test, though, so my prime suspect is environmental. | |
| 14:55:53 | bnemec | Reentrancy has to do with concurrent execution, which you can't get in a single thread. | |
| 14:56:06 | mdbooth | bnemec: You might want to look that up ;) | |
| 14:56:17 | bnemec | I did. Did you? https://en.m.wikipedia.org/wiki/Reentrancy_%28computing%29 | |
| 14:56:25 | bnemec | "In computing, a computer program or subroutine is called reentrant if multiple invocations can safely run concurrently." | |
| 14:56:39 | mdbooth | https://docs.python.org/3/library/threading.html#semaphore-objects | |
| 14:57:07 | mdbooth | https://en.wikipedia.org/wiki/Reentrant_mutex | |
| 14:57:52 | bnemec | Ah, we're talking about different things. | |
| 14:59:35 | mdbooth | So, Semaphore doesn't appear to be mocked, and yet I see 2 enters followed by 2 exits | |
| 15:00:36 | openstackgerrit | Eric Fried proposed openstack/nova stable/train: Func: bug 1849165: mig race with _populate_assigned_resources https://review.opendev.org/690099 | |
| 15:00:36 | openstack | bug 1849165 in OpenStack Compute (nova) "_populate_assigned_resources raises "TypeError: argument of type 'NoneType' is not iterable" during active migration" [High,In progress] https://launchpad.net/bugs/1849165 - Assigned to Eric Fried (efried) | |
| 15:01:20 | mdbooth | bnemec: To be clear, I'm *not* expecting to find a bug in oslo.concurrency. I'm expecting to find a fixture somewhere doing something I haven't thought of yet. | |
| 15:03:17 | mriedem | smells like a bro down in here https://www.youtube.com/watch?v=CHnbC1N_G6w | |
| 15:05:49 | mdbooth | Oh, great, it's a heisenbug >:( | |
| 15:07:39 | adrianc | sean-k-mooney, i can do the backport for https://review.opendev.org/#/c/675776/, however it will take me some time to get a setup up to test it | |
| 15:08:05 | bnemec | mdbooth: By any chance are you running tests in parallel? The external lock fixture from oslo.concurrency would make the locks independent. | |
| 15:09:36 | openstackgerrit | Dan Smith proposed openstack/nova master: Add image precaching docs for aggregates https://review.opendev.org/687348 | |
| 15:11:17 | openstackgerrit | Eric Fried proposed openstack/nova stable/train: Don't populate resources for not-yet-migrated inst https://review.opendev.org/690100 | |
| 15:11:39 | efried | there must be an easier way to cherry pick a series | |
| 15:15:23 | sean-k-mooney | efried: cerry picking a serise is baically a rebase | |
| 15:15:31 | sean-k-mooney | do you mean for backporting? | |
| 15:15:50 | efried | yes | |
| 15:15:55 | sean-k-mooney | ah | |
| 15:16:04 | sean-k-mooney | i was refering to a feature branch before | |
| 15:16:50 | sean-k-mooney | there is proably a way to do it in like 1 comamnd in git but i dont know what it is | |
| 15:17:48 | sean-k-mooney | oh git cherry-pick takes a list of commit | |
| 15:18:46 | sean-k-mooney | so if you do a git log betwen the base of the serise and top and just get the commits you shuld be able to pipe the revers of that to reverse and xargs to invoke the cherry pick | |
| 15:21:06 | efried | will that pile them up in a series? | |
| 15:21:15 | efried | that's probably the ticket, then. I'll try that next time. | |
| 15:21:29 | sean-k-mooney | i think so also looking at the exmaple in the git man pages | |
| 15:21:36 | sean-k-mooney | you might be able to do somting like this | |
| 15:21:46 | sean-k-mooney | git cherry-pick start..end | |
| 15:21:57 | dansmith | can't you do a range? | |
| 15:21:57 | sean-k-mooney | where start and end are two comites | |
| 15:21:58 | dansmith | yeah that | |
| 15:22:03 | dansmith | works for any other git command that takes multiple commits | |
| 15:22:31 | dansmith | if you have them all in a branch, then something like "git cherry-pick -x branch~4.." probably works | |
| 15:22:31 | sean-k-mooney | they have an example fo "git cherry-pick ..master" | |
| 15:22:48 | sean-k-mooney | that cherrypick anything on master but not locally | |
| 15:22:54 | sean-k-mooney | so i assuem you can close the range | |
| 15:23:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Require Migration object arg to migrate_instance_finish method https://review.opendev.org/690106 | |
| 15:24:49 | umbSublime | I have a specific workload where I need to enable invTSC, for this to work with migrations I also need to set the tsc frequency. How can I configure the tsc frequency in nova.conf on my compute hosts (is it possible with cpu_model_extra_flags)? in qemu it looks like this: "<clock><timer name='tsc' frequency='3000000000'/></clock>" | |
| 15:25:25 | sean-k-mooney | umbSublime: im pretty sure you cant | |
| 15:26:02 | sean-k-mooney | you could enabel a specific cpu flag but i dont think you can spcify the frequency | |
| 15:29:06 | umbSublime | Would the solution in this case be to create a custom CPU model and use that in nova.conf ? | |
| 15:29:16 | umbSublime | or is there an alternative workaround I didn't think about | |
| 15:29:54 | sean-k-mooney | well this is really a feature request so i would start with the uscase | |
| 15:30:33 | sean-k-mooney | you are trying to fake a specifc cpu frequece and provide a "stable" tsc to the guest correct? | |
| 15:30:39 | umbSublime | I'm only the operator for this cloud. But the use case is a software that requires that flag. Only issue is with only the flag enbaled and the frequency unset, migrations are not possible | |
| 15:30:43 | umbSublime | due to the nature of the flag | |
| 15:30:47 | sean-k-mooney | so that if you migrate betwwen hosts with differen cpu fequecne it does not chagne | |
| 15:31:10 | umbSublime | exactly there are some extensive discussions about that and how it was implemented in qemu/libvirt | |
| 15:31:51 | umbSublime | when you set a specific frequency, there is a calculation made so that migration is possible and the value of the register is mirrored over | |
| 15:32:44 | sean-k-mooney | ya so nova dose not currently suport hat | |
| 15:32:48 | sean-k-mooney | *that | |
| 15:32:51 | umbSublime | All I know is the software they run on the guest requires that invTSC is presented to the guest. From my operation pov I also need the VM to be migratable | |
| 15:33:00 | sean-k-mooney | we could ad support if we had a defied usecase | |
| 15:33:28 | sean-k-mooney | do they need invTSC set or do the want the TSC to be marked as reliable in the vm | |